New Proposal for Review Process & Delightful Add-ons

<<

fligtar

Add-ons Product Manager

Posts: 132

Joined: Tue Sep 22, 2009 8:00 am

Location: SF Bay Area

Post Fri May 21, 2010 3:55 pm

New Proposal for Review Process & Delightful Add-ons

We previously proposed an Incubation Process as a way to make add-ons hosted on AMO safer while still letting developers host their experimental add-ons via direct links. Developer feedback in these forums raised several issues:

  • developers value the feedback given by random users that discover their add-ons in the sandbox
  • developers don't want to feel rushed in the 30 (or 90) day time window with which they must pass review
  • developers consider the review process a significant milestone and are not all willing to assume that responsibility

Taking these into account, we've come up with a new plan that accomplishes our previous goal of making all add-ons publicly available on AMO safe, and also provides incentives for developers to make great add-ons.

Please take a look and let us know what you think of the new proposal.
<<

mozjonathan

Posts: 3

Joined: Tue May 11, 2010 11:56 pm

Post Fri May 21, 2010 4:16 pm

Re: New Proposal for Review Process & Delightful Add-ons

I think that's a great proposal. It's much clearer and much easier to understand than the previous proposal, and I love the idea of delightful addons. It will raise the overall quality of the addons with even more incentives for developers to make great addons, and honestly, there's no way anyone serious about developing addons can't pass the "preliminary review" stage. This will also allow for a huge cleanup of the addons database.

Kudos!
<<

webgapps

Posts: 100

Joined: Sun Jan 17, 2010 5:17 pm

Post Fri May 21, 2010 5:33 pm

Re: New Proposal for Review Process & Delightful Add-ons

I have some concerns about this:

When a new add-on is submitted to AMO, it is called "unverified" and will first go through automatic validation and virus/malware scanning. Any serious problems detected will require resolution before the process can continue (the add-on is rejected without its entry even being created);


I have a couple of add-ons that were flagged by the automatic validation, but approved by the editor. So let's say the validation detects a false positive, then what can I do if my add-on won't even be added to the gallery? I still prefer that my add-on is not viewable by anyone else besides me and the editors, than not being able to add it at all.

On a side note, I kind of liked the incubation proposal, because it has already pushed me into fixing bugs and nominating some add-ons that otherwise would stay as experimental for a much longer time.
<<

jorge-villalobos

AMO Administrator

Posts: 3005

Joined: Tue Sep 29, 2009 7:30 pm

Location: San José, Costa Rica

Post Fri May 21, 2010 5:44 pm

Re: New Proposal for Review Process & Delightful Add-ons

All warnings you currently see in the validator are just warnings. They won't prevent you from submitting the add-on. Only serious errors (like a malformed add-on package) and malicious code will.
<<

kinger

AMO Editor

Posts: 3

Joined: Wed Sep 30, 2009 1:08 pm

Location: Slovenia

Post Thu Jun 03, 2010 7:14 am

Re: New Proposal for Review Process & Delightful Add-ons

The proposal does not really say what full review involves. Given that code examination is in the preliminary review, I presume it means functional and UX testing. I think having two reviews will provide unnecessary overhead to editors.

I can see what you are getting at with the term 'Delightful', but I don't think it is appealing. I see some alternatives listed in the doc. It would be a good idea to avoid US-centric terms such as awesome.
<<

eviljeff

AMO Senior Editor

Posts: 317

Joined: Wed Sep 30, 2009 12:40 am

Location: Sheffield, UK

Post Thu Jun 03, 2010 1:33 pm

Re: New Proposal for Review Process & Delightful Add-ons

Its an improvement on the previous iteration of the proposal and should provide a better balance.

I'd like a more detailed explanation of what will and won't be checked in the preliminary review really. Full code review takes at least 50% of the review time (with some large addons it can be a lot more) - if all the code has to be checked to check for malicious code (but minor policy violations ignored I presumed) then the 'quick' review is going to take nearly as long as a current style review. If we're only checking for maliciousness on the preliminary review then for the full review the code will have to be checked again I presume, plus testing and some subjective evaluation like now.

How are self hosted addons going to fit into the process? Are they going to require any kind of preliminary review to be consistent?

p.s. while we're at it I'd like if it we could move away from the word 'review' as I think it can get confused with user reviews. Submission? Its always seemed a little inconsistent to change from Reviewer to Editor with Remora but keep calling them reviews.
<<

rkentjames1

AMO Editor

Posts: 1

Joined: Sat Jun 05, 2010 7:48 am

Post Sat Jun 05, 2010 8:00 am

Re: New Proposal for Review Process & Delightful Add-ons

Overall I like the new proposal. I have used experimental addons to either help developers or test new features, and 30 days was too short. This new proposal allows a middle path.

Just a caution though. I've designed a lot of business processes, and the natural tendency is to add complexity in an attempt to more precisely match all of the different scenarios that are possible. The more complex proposals often sound great, but then are discovered to be not worth the extra bureaucracy. I feel that this proposal is risking moving in that direction.

One suggestion that I would make is to give the editor the option of doing a different review than requested. That is, for a simple extension, if the user requested preliminary review but I as a reviewer can see that I can easily do a full review, I should be able to do that directly. Similarly, if the user requested a full review, and the extension meets the prelimininary review but not the full review criteria, then I should be able to give the preliminary review approval while we are still iterating on getting the full review done, like if there are some issues. That might complicate automatic updates though.

rkent
<<

patrickjdempsey

Posts: 53

Joined: Thu Oct 01, 2009 3:57 pm

Post Sun Jun 13, 2010 7:33 pm

Re: New Proposal for Review Process & Delightful Add-ons

@rkent, I think you may be missing a part of the point of this. One of the major problems with the previous proposal is that it took control away from the developer in terms of when an add-on goes public. It's very important to the developer that the add-on goes public when he thinks it's ready, not two months later (a problem AMO had in the past), and not at any moment before (as you propose). The point of the Preliminary Review track (as I understand it) is that it gives developers the opportunity to use AMO as a test platform. In my opinion, this is absolutely crucial to maintaining a high level of quality in add-ons, as well as cultivating an environment that welcomes new developers. The previous proposal focused on security and forgot about development. By splitting the sandbox into a "closed sandbox" for the Preliminary Review queue and an "open sandbox" for testing before the Final Review, I think they've found a nice balance between security and development.

In general I think this proposal is really good. It addresses my (and I'm sure many other's) concerns well. My only comments are that I don't care much for "Delightful Add-ons" and the other suggested names all sound really dated. Time to crack open the Thesaurus. ;) The other is that the 10,000 daily users limit for the Delightful Add-ons will probably hurt Themes. There is currently a dramatic decline in Use statistics for Themes which is puzzling many developers:

https://bugzilla.mozilla.org/show_bug.cgi?id=566124

I understand the reasoning behind the use limit, but I just wanted to make sure you guys are aware that there's something odd going on with those numbers right now.
<<

road.runner

Posts: 21

Joined: Tue May 11, 2010 10:09 am

Post Tue Jun 29, 2010 9:33 am

Re: New Proposal for Review Process & Delightful Add-ons

Justin Scott (fligtar) wrote:We previously proposed an Incubation Process as a way to make add-ons hosted on AMO safer while still letting developers host their experimental add-ons via direct links. Developer feedback in these forums raised several issues:

  • developers value the feedback given by random users that discover their add-ons in the sandbox
  • developers don't want to feel rushed in the 30 (or 90) day time window with which they must pass review
  • developers consider the review process a significant milestone and are not all willing to assume that responsibility

Taking these into account, we've come up with a new plan that accomplishes our previous goal of making all add-ons publicly available on AMO safe, and also provides incentives for developers to make great add-ons.

Please take a look and let us know what you think of the new proposal.


I agree... the initial proposal would have been severely restrictive to developers, something which might be acceptable for closed source companies like Apple, but doesn't suit FOSS products like Firefox.
Last edited by road.runner on Tue Jun 29, 2010 9:53 am, edited 1 time in total.
<<

road.runner

Posts: 21

Joined: Tue May 11, 2010 10:09 am

Post Tue Jun 29, 2010 9:48 am

Re: New Proposal for Review Process & Delightful Add-ons

I have another suggestion too...

I think that the message below the download tag should be set by the addon reviewer, and reflect the actual status of the add-on.

For eg,
Let the default status message for an add-on that a developer has not requested for review, state the same fact... "This add-on has not been reviewed by Mozilla"
But when a add-on reviewer rejects an add-on from public listing, he/she always informs the add-on developer about the reason for retaining the add-on in sandbox. I think this information is really, really useful, and should be made available as the status message of the add-on below the "Download Now" button. Because informing AMO visitors to an add-on page that "This add-on has not been reviewed by Mozilla" while in reality, the add-on has actually been reviewed (and found to be faulty or not worthy of public listing) is saying way too little and actually false in a sense.


Please instead inform something, like:
"This add-on has been not been approved as a public addon by Mozilla, because it has...
1) loose JS variables and can cause a conflict with other other add-ons, or...
2) we feel that this add-on is meant for a limited audience and has a very small active users base as of now, or...
3) etc, etc. (The add-on editors and developers can fill up more such reasons here, based on their experience ;-) )

Coming to an actual example, just check out the info below. It communicates so much info that would be highly valuable in giving a better picture of the consequences of installing an addon to an add-on user:

Comments: Dear add-on developer,
It is unclear to us at this time if your add-on will be useful for a general audience so that it warrants public listing. We're rejecting your nomination, but encourage you to promote your listing so that it garners more active users and user reviews. You may re-nominate your add-on once there is more proof of its usefulness.
<<

archaeopteryx

AMO Editor

Posts: 1036

Joined: Fri Sep 25, 2009 1:38 pm

Location: Dresden, Germany

Post Wed Jun 30, 2010 1:13 am

Re: New Proposal for Review Process & Delightful Add-ons

<<

tonymec

Posts: 7

Joined: Tue Jul 13, 2010 4:51 pm

Location: Brussels, Belgium

Post Tue Jul 13, 2010 5:08 pm

Re: New Proposal for Review Process & Delightful Add-ons

Remark 1.
Criteria for applying to the Delightful Add-ons Program:
[...]
  • Must be compatible with the latest stable Firefox release at time of launch
[...]

I don't see why a MailNews-only add-on (supporting Thunderbird and/or SeaMonkey but not Firefox) or even a Calendar add-on (supporting Sunbird, or now that Sunbird is defunct, Thunderbird+Lightning and/or SeaMonkey+Lightning) could not belong to "the best of the best". Therefore I propose instead:
  • Must be compatible with the latest stable release of all concerned applications at time of launch

Remark 2.
«Delightful add-ons» sounds dated to me, also subject to the whims of fashion. I believe that we should go for a more neutral label, but one that could withstand the proof of time. I suggest: Officially featured add-ons (or maybe just "Featured add-ons"), or else: The Best of the Best.
<<

customfirefoxlady

Posts: 53

Joined: Wed Oct 28, 2009 5:22 am

Post Thu Aug 12, 2010 5:38 am

Re: New Proposal for Review Process & Delightful Add-ons

Any known time-frame for a decision on whether/what/when something like this might get implemented?
<<

jorge-villalobos

AMO Administrator

Posts: 3005

Joined: Tue Sep 29, 2009 7:30 pm

Location: San José, Costa Rica

Post Thu Aug 12, 2010 7:51 am

Re: New Proposal for Review Process & Delightful Add-ons

This spec will be implemented either this quarter or later this year, but no later than that. It will be implemented as specified in the document, unless there are new points brought up for discussion.
<<

scripter1

Posts: 8

Joined: Mon Jan 10, 2011 2:41 am

Post Mon Jan 10, 2011 3:04 am

Re: New Proposal for Review Process & Delightful Add-ons

Criteria for applying to the Delightful Add-ons Program:
  • Must have an active developer who provides support channels for the add-on and responds to bugs quickly
  • Must be compatible with the latest stable Firefox release at time of launch
  • Must have at least 10,000 active daily users and positive weekly trends
  • Must be delightful, as explained above

How about SeaMonkey or Thunderbird?
Many addons are SeaMonkey specific (like addons adding some of Firefox's functionality to SeaMonkey that are not implemented internally) or Thunderbird specific.
Thunderbird have less users than Firefox and SeaMonkey have even much less users than Firefox andThunderbird (please, correct me if I'm wrong).

I propose to change slightly criteria like:
  • Must have an active developer who provides support channels for the add-on and responds to bugs quickly
  • Must be compatible with the latest stable release of applications that it works with at time of launch
  • Must have at least 10,000 active daily users (or at least 7,000 active daily Thunderbird users if it is Thunderbird specific or at least 4,000 active daily SeaMonkey users if it is SeaMonkey specific) and positive weekly trends
  • Must be delightful, as explained above

I think that would be good idea to separate Delightful Add-ons for Firefox, for Thunderbird and for SeaMonkey.
<<

tonymec

Posts: 7

Joined: Tue Jul 13, 2010 4:51 pm

Location: Brussels, Belgium

Post Mon Jan 10, 2011 8:43 pm

Re: New Proposal for Review Process & Delightful Add-ons

scripter1 wrote:[...]
I think that would be good idea to separate Delightful Add-ons for Firefox, for Thunderbird and for SeaMonkey.

Maybe when listing them or searching for them, the way it is already done now at AMO. But OTOH (and this will have to be taken into account when "crowning" a new Delightful Add-On), I believe that the following "classes" of addons are (or should be) all quite well-populated:
  • Firefox only
  • Thunderbird only
  • SeaMonkey only
  • Browser (Firefox and SeaMonkey)
  • MailNews (Thunderbird and SeaMonkey)
  • Any Toolkit app (including "Gecko" addons for any app with addon-manager support)
  • and maybe Other
BTW, wouldn't it be nice if "bonus points" (or something) could be awarded for addons supporting more than one application (and, of course, supporting them all "delightfully")?

Oh, and what happens if some "formerly delightful" addon fails to support the next stable release once it is released as stable (e.g. because the add-on author has gone AWOL)? Delisting, I suppose, but immediate (once the public release succeeds to the last beta), or with some grace period?
<<

fligtar

Add-ons Product Manager

Posts: 132

Joined: Tue Sep 22, 2009 8:00 am

Location: SF Bay Area

Post Mon Jan 10, 2011 8:58 pm

Re: New Proposal for Review Process & Delightful Add-ons

Just an update on delightful add-ons... the new review process launching this week doesn't introduce them. We decided to leave them out and don't have any current plans for implementing them in the next few months.
<<

tonymec

Posts: 7

Joined: Tue Jul 13, 2010 4:51 pm

Location: Brussels, Belgium

Post Tue Jan 11, 2011 8:19 am

Re: New Proposal for Review Process & Delightful Add-ons

fligtar wrote:Just an update on delightful add-ons... the new review process launching this week doesn't introduce them. We decided to leave them out and don't have any current plans for implementing them in the next few months.

Oh, so "project shelved", is that it? Suspended sine die? Looked interesting though. :-( Well, I suppose we'll have to go by ratings, #downloads, "Featured", etc., the way they are already implemented at AMO.
<<

ThunderBird303

Posts: 5

Joined: Tue Feb 15, 2011 1:16 pm

Post Tue Feb 15, 2011 1:27 pm

Re: New Proposal for Review Process & Delightful Add-ons

fligtar wrote:We previously proposed an Incubation Process as a way to make add-ons hosted on AMO safer while still letting developers host their experimental add-ons via direct links. Developer flashlights in these forums raised several issues:

  • developers value the feedback given by random users that discover their add-ons in the sandbox
  • developers don't want to feel rushed in the 30 (or 90) day time window with which they must pass review
  • developers consider the review process a significant milestone and are not all willing to assume that responsibility

Taking these into account, we've come up with a new plan that accomplishes our previous goal of making all add-ons publicly available on AMO safe, and also provides incentives for developers to make great add-ons.

Please take a look and let us know what you think of the new proposal.


What I have always found a bit frustrating was the thirty day window. As a developer, it is important to take time and make sure you are releasing a well-operating add on for this browser. I know of many developers who have been loyal to Firefox and are now gravitating towards Chrome for just this reason. I have not been in the business for a couple of years now, but if there is still such a tight time frame for passing review, I may consider moving forward if I return to the game. This is an old thread and things may have changed, so I apologize if I may be mistaken.

Return to Announcements and Roadmap

Who is online

Users browsing this forum: No registered users

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
CA Gen2 style designed by Vjacheslav Trushkin.