Thursday, August 30, 2012

Greasemonkey API Usage -- August 2012

Back in November 2009, I analyzed the API usage, and a few other aspects, of all the scripts on userscripts.org, then 36,141 scripts.  I was directly discussing some of the topics that were already bubbling around the back of our minds, for how to carry Greasemonkey into the future with us.  The short version is: web browsers and web apps are getting so much poweful, why do we need these one-off Greasemonkey APIs with cross-browser problems?

Now that Greasemonkey 1.0 is out, we've made big steps made in that direction, and I've repeated the analysis.  I downloaded (with permission from the site owner) every single active script on userscripts.org, now 82,084 scripts.

First up, which API calls are made, and how common are they?
API Usage by Number of Scripts

Not much has changed.  By far most common is that a script doesn't call any special APIs (57.94%).  Then, GM_getValue/GM_setValue are still right up there.

The biggest change is that unsafeWindow usage has jumped from 6th to 2nd place (12.44% to 17.65% of scripts; 1,527 scripst also mention wrappedJSObject, not on the chart).  Authors want to interact with the page in ways that the security sandbox (which protects these APIs) prevent, so they explicitly jump out of the sandbox, bringing vulnerabilities with them.  Of the 14,484 scripts that reference unsafeWindow, 6,494 of them use no other Greasemonkey APIs, and thus are served well by moving towards a model where there is no sandboxing (nor APIs, unless you ask for it/them).  Another 838 scripts only use GM_log and/or GM_addStyle, which can easily be replaced with console.log() or the compatibility shim layer.  It gets hard to analyze other calls in more detail, but I see a lot of get/set value calls, which (assuming you run on only one domain) can also be well served by DOM Storage.

Moreover, those 47,557 scripts that don't use any of the special APIs are still saddled with the sandbox and its pitfalls, known and unknown before Greasemonkey 1.0.  Plenty of newer browser features don't work in the security sandbox because its entire point is to separate the content scope (where these features work) from the script scope (with its smaller set of privileged features).  A huge part of the design changes in Greasemonkey 1.0 is to make the default behavior, like these majority of scripts need/want, to run as close to possible as a regular script in a regular web page, without surprises like missing values and broken features.

So do scripts that use get/set value or xmlhttpRequest really need the cross-domain behavior they provide?
API Usage Cross-domain Analysis
(Note: the left-most set of bars is "@include *" and the rightmost is ">5" -- the labels are missing from the graph and I'm not sure why.)
Mostly: no, and this hasn't changed much since 2009.  The vast majority of scripts using get/set value (71.86%) only ever execute on a single domain, and thus can use DOM Storage with no ill effects.

The XHR usage to two domains is lower mostly because I fixed my analysis a bit (i.e. not counting an @include of *.example.com and an XHR to www.example.com as two domains, and not counting XHRs to userscripts.org, assumed to be update checker scripts, which is now provided by Greasemonkey).  However, a combined 44.25% of scripts that call XHR (and with a string literal that I could pull a domain name out of, not a variable set somewhere else) either call to/run on two or all domains, and thus really use the cross-domain power of GM_xmlhttpRequest.

Finally a bit more detail about Metadata imperatives.  This graph is for all imperatives used in at least 1% of scripts, regardless of what they are.
Greasemonkey Metadata Imperative Usage
Most of what has changed since 2009 is the analysis, including more values.  Note that almost every script (99.37%) specifies @name, and we see a power law trail off in usage.  The commonly used, but unsupported in Greasemonkey, ones are @author, @homepage, @license/@copyright, @date, and @history.

Check the raw data to see hundreds more @things, generally all unsupported values.  And there I pasted only those used at least ten times, there are yet more hundreds used fewer times.

To those that are interested: the script that I used to generate these numbers is available for inspection, in case it perhaps contains a serious bug. The raw data that I generated with it, and the charts above, are also available to check.

Tuesday, August 28, 2012

Greasemonkey 1.0 + jQuery: Broken, with Workaround

One of the big changes behind Greasemonkey 1.0 was moving towards the goal of not forcing the security sandbox (and all its pitfalls) upon script authors.  This is the entire reason for @grant, and specifically the @grant none setting.  In the @grant none case, the script does not get the traditional security sandbox (with XPCNativeWrappers), but rather a very thin sandbox that exclusively acts as a private scope, to hold variables for the script that don't interact with the page.

The idea was, if you want to set something in the page, you just do window.foo = 'bar', and if you don't it's just a normal var foo = 'bar'.  But there's a problem.

If you @require jQuery, it implicitly does a window.$ = window.jQuery = ...,  which exports the copy of jQuery that your script is loading into the page.  If they're different versions, there is a very real possibility of completely breaking the page.

This is Greasemonkey issue 1614, which is open and being tracked for a fix.  In the meantime, you can insert a one line fix into your script, at the top level (not inside any functions):

this.$ = this.jQuery = jQuery.noConflict(true);

This line just calls the standard jQuery noConflict() method, so that this loaded version of jQuery doesn't conflict with anything already in the page.  It's already there, it's exactly what it's for!  And saves a local (in the script) reference to the version of jQuery that you want to use.

This should let your script keep working, and also keep the page from breaking.  It's only lightly tested so far; let us know in the comments if it helps you.

Friday, August 24, 2012

Greasemonkey 1.0 Release

After more than seven years, Greasemonkey has finally grown to version 1.0.

Back in August of 2005 (almost exactly seven years ago now), Greasemonkey introduced wrappers intended to plug security holes.  As a result the common pitfalls were born.  Ever since then, in order to write a user script that would function properly in Greasemonkey, authors were required to either get lucky and not trip over one of these pitfalls, or get lucky and figure out that they exist -- and how to work around them.

As of today, all you have to know is "@grant none".  If you specify this setting in your metadata, then none of these security wrappers are put around your script. And you aren't granted access to any of the Greasemonkey APIs that a normal page wouldn't have.  Almost anything you can do in a script in the page itself should work in a "@grant none" user script.  There is still a sandbox; this isolates your variable scope from the page itself.  Without this it is extremely easy to break the page.  In order to explicitly read/write to/from the content scope, just reference properties of the window object.  (I.E. "x = 10" assigns to a variable x only in the user script's private scope.  "window.x = 10" assigns to the variable x in the content page's scope, which it can see.)


The entire list of bugs handled in this release is also available via the 1.0 milestone on GitHub. A number of issues listed there only affected Greasemonkey during the development of version 1.0, so they aren't listed as changes below.

As always, if you notice problems, it's best to log an issue at GitHub or let us know at the greasemonkey-dev mailing list (and be clear that your issues are with this version).

Enhancements since Greasemonkey 0.9.x:
  • New metadata, @grant, specifies which special APIs a user script will have access to.  Specifying @grant none means no special API access, and thus no security restrictions.  Then, everything you're used to doing in JavaScript in a web page (including but not limited to jQuery) should just work.  For legacy scripts (which have no @grant line at all), Greasemonkey will try to guess what @grant lines you should have.  See http://wiki.greasespot.net/@grant for more detail. (#1425, #1427, #1558)
  • The toolbar button is colorful (in the enabled state) on Mac OS X. (#1597)
  • The metadata @unwrap has been removed, as being unwrapped is now the default.  The wrapper will still be applied to scripts that have a "return" statement outside of any function, but this may be removed in the future, so make sure your scripts (and requires) don't do this; authors may manually add an anonymous function wrapper around the script for the exact same behavior. (#1568, #1592)
  • Scripts that @run-at document-start have a valid document object to modify, E.G. for adding <style> tags; but still before any part of the document is loaded. (#1565)
  • GM_xmlhttpRequest() accepts a timeout option.  (#1561)
  • GM_getResourceURL() works with a special protocol handler.  (This is more efficient/faster than the data: URI encoding used previously.)  For example, specify images and styles with URLs to your @resources.
  • The standard Firefox web developer console works for console.log() et al.  (#1564)
  • Automatic updates work correctly with scripts installed from userscripts.org (but still note the require secure updates setting). (#1555)
  • Require at least Firefox 14.0 (no more Firefox 3 compatibility).  (#1426, #1522)
  • Error reporting is much more consistent and obvious than in the past. (#1404, #1592)
  • The alert() workaround (see http://bugzil.la/647727) is not applied for Firefox versions that do not exhibit this bug.  (#1318, #1350)
Bug fixes since Greasemonkey 0.9.x:
  • When downloading a script not encoded in UTF-8, display an error message to the user (rather than just failing). (#1588)
  • The "show script" button in the install dialog is disabled until the download of the script file is complete. (#1586)
  • Scripts with missing or broken "==UserScript==" metadata will work.  (#1562)

Thursday, August 09, 2012

Beta: Greasemonkey Release 1.0beta7

The entire list of bugs handled (and some still pending) in this release is also available via the 1.0 milestone on GitHub. This is only a beta release so you'll need to head to the all versions page to find it.

After more than seven years, Greasemonkey is finally graduating to version 1.0!  We're taking the major version number bump as an opportunity to reconsider some big ideas.  As of right now we believe there are appropriate detections and modes to make everything continue to work as always, but we're laying the groundwork to really break backwards compatibility, perhaps in a 1.1 release.

Keep an eye on this blog for posts dedicated to more detail on these topics.  For now each is briefly mentioned in the changelog below.

We desperately want feedback on this beta release, especially from script authors.  If you are using it and notice problems of any kind or have any other feedback for us, it's best to log an issue at GitHub or mail us at greasemonkey-dev (and be clear that which version is under discussion).

Enhancements since Greasemonkey 0.9.x:
  • New metadata, @grant, specifies which special APIs a user script will have access to.  Specifying @grant none means no special API access, and thus no security restrictions.  Then, everything you're used to doing in JavaScript in a web page (including but not limited to jQuery) should just work.  For legacy scripts (which have no @grant line at all), Greasemonkey will try to guess what @grant lines you should have.  See http://wiki.greasespot.net/@grant for more detail. (#1425, #1427, #1558)
  • The metadata @unwrap has been removed, as being unwrapped is now the default.  The wrapper will still be applied to scripts that have a "return" statement outside of any function, but this may be removed in the future, so make sure your scripts (and requires) don't do this; authors may manually add an anonymous function wrapper around the script for the exact same behavior. (#1568, #1592)
  • Require at least Firefox 14.0 (no more Firefox 3 compatibility).  (#1426)
  • Scripts that @run-at document-start have a valid document object to modify, E.G. for adding style tags; but still before any part of the document is loaded. (#1565)
  • GM_xmlhttpRequest() accepts a timeout option.  (#1561)
  • GM_getResourceURL() works with a special protocol handler.  (This is more efficient/faster than the data: URI encoding used previously.)  For example, specify images and styles with URLs to your @resources.
  • The standard Firefox web developer console works for console.log() et al.  (#1564)
  • Error reporting is much more consistent and obvious than in the past. (#1404, #1592)
Bug fixes since Greasemonkey 0.9.x:
  • Scripts with missing or broken "==UserScript==" metadata will work.  (#1562)
  • The alert() workaround (see http://bugzil.la/647727) is not applied for Firefox versions that do not exhibit this bug.  (#1318, #1350)
  • When downloading a script not encoded in UTF-8, display an error message to the user (rather than just failing). (#1588)
  • The "show script" button in the install dialog is disabled until the download of the script file is complete. (#1586)

Tuesday, July 31, 2012

Greasemonkey 0.9.22 Release

The entire list of bugs handled in this release is also available via the 0.9.21 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

This release contains just one important fix:

Bug fixes:
  • Fix memory leak related to GM_registerMenuCommander. (#1578)
Update:
I bungled a bit of the release, as early commenters below note.  I also hit a bug in the Mozilla Add-ons site in the process, so this release is now called 0.9.22, and officially there is no version 0.9.21.  Sorry!

Monday, May 14, 2012

Greasemonkey 0.9.20 Release

The entire list of bugs handled in this release is also available via the 0.9.20 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

Enhancements:
  • When the toolbar is in text-only mode, the "Greasemonkey" label on the toolbar button will be striked-out when Greasemonkey is disabled. (#1544)
  • When selecting a script editor, the previous choice will be remembered and displayed. (#1546)

Bug fixes:
  • A change in 0.9.19 broke setTimeout() for a variety of cases, that has been reverted. (#1549, #1552, #1553)
  • Scripts with long names and/or file names could fail to install in Windows. (#1548)

Monday, April 23, 2012

Greasemonkey Release 0.9.19

The entire list of bugs handled in this release is also available via the 0.9.19 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

Enhancements:
  • A custom wrapper to make setTimeout() work even when Javascript is disabled. (#1209)
  • Restored the "script installed successfully" toast notification. (#1511)
  • Scripts download more incrementally; the install dialog should appear as soon as the script metadata has been downloaded, not after the entire script is downloaded.  (#1523)
Bug fixes:
  • Specifying a script editor in a non-ASCII folder will work (Firefox 4+ only). (#1173)
  • Scripts installed while the Add-Ons Manager is open will show their icon. (#1535)
Plus a variety of minor code cleanups and enhancements behind the scenes.

Tuesday, February 28, 2012

Greasemonkey 0.9.18 Release

The entire list of bugs handled in this release is also available via the 0.9.18 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

Enhancements:
Bug fixes:
  • Add-on Manager shows correct data when re-installing (e.g. to update) a script. (#1476)
  • Re-installed scripts are put into the same directory as the original. (#1515)
  • The matches property of GM_info is now a list of (pattern) strings. (#1516)

Automatic script updates come to Greasemonkey

In the 0.9.18 release, Greasemonkey is now checking for, and installing, updated versions of user scripts by default.  This post is intended as a primer for how this new functionality works, both for users and for script authors.


The Greasemonkey Options dialog now includes these settings to control automatic update checking and installation. By default updates will be checked for every seven days, and automatically installed when found.  The download location must be secure (https) by default.  And it should just work!

If you know there's an update, you can open the Add-ons manager, right click on a User Script, and choose "Find Updates".  This will check immediately, even if a scheduled check has happened more recently than seven days ago.*

*You may need to clear your browser cache, if an old version of the script was downloaded recently.

Script Authors
Greasemonkey supports a number of new metadata directives to support updates: @downloadURL, @updateURL and (more than before) @version.

The @updateURL will be accessed when checking for the update.  This can be a ".meta.js" format like userscripts.org produces, with only the metadata.  The @version defined here will be checked to the install version.  It does not need to be hosted on https to be checked by default.

Sidebar: for userscripts.org, the updateURL is automatically set to the .meta.js corresponding to the script, and passed through the Coral Content Distribution Network (see why).

The @downloadURL is where the update will be downloaded from, when found.  This is most useful to guarantee that updates come from a secure (https) location, so that they will be installed by default.

If @updateURL or @downloadURL are not provided, the URL that the script was downloaded from is used instead.  If the @downloadURL is not provided and the install URL is not known, updates will never be applied.

Finally the @version directive controls what is considered an update.  Read about the (Mozilla) toolkit version format to learn exactly what value is greater than what, and what formats are valid.

It's also worth pointing out that the recently added GM_info API can be used to check if Greasemonkey will ever try to apply updates (e.g. because the user has turned it off, or the URL is insecure).

Finally, if you want to make sure that Greasemonkey will never update your script, it can be accomplished by providing an invalid @updateURL (like "about:blank") which will never return a newer version.

Questions?
Please bring any questions or comments to us at the greasemonkey-users discussion list.

Saturday, February 11, 2012

Greasemonkey 0.9.17 Release

Whoops, there was a problem with 0.9.16 yesterday, so here's a quick patch.

Bug fixes:
  • Error: "aForced is not defined".  (#1517)

Friday, February 10, 2012

Greasemonkey 0.9.16 Release

The entire list of bugs handled in this release is also available via the 0.9.16 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

Enhancements:
  • Expose information (including Greasemonkey version) to user scripts via GM_info. (#1452; also see #1512)
  • Track the install time of scripts.  (This is designed to better support automatic updates for scripts ... still coming soon.)  (#1513)
Bug fixes:
  • Editing script from the "New User Script" feature, to change any remote dependency (@icon, @require, @resource), could the script to malfunction. (#1504)
  • Enabling/disabling scripts from the status bar menu (Firefox 3 only). (#1506)
  • Incompatibility with GreaseFire.  (#1507)
  • Failures during "Check for Updates" from the Add-on manager. (#1509)

Wednesday, January 25, 2012

Greasemonkey 0.9.15 Release

The entire list of bugs handled in this release is also available via the 0.9.15 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

This release was hurried, to resolve a few serious bugs introduced in 0.9.14. A few other bugs were also fixed.

Bug fixes:
  • Editing an installed script to change any remote dependency (@icon, @require, @resource) could cause one or many scripts to be deleted. (#1466, #1502)
  • Certain @requires can break a script, due to javascript semicolon insertion rules.  (#1491)
  • Toggling the enabled state of a script via the main (Tools > Greasemonkey) menu failed on Mac.  (#1496)
  • Installing a script from local disk failed in 0.9.14.  (#1501)

Friday, January 20, 2012

Greasemonkey 0.9.14 Release

The entire list of bugs handled in this release is also available via the 0.9.14 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

Enhancements:
  • For new script installs, and installed scripts' updates (still not enabled by default) there is a progress meter for the download. (#1419)
Bug fixes:
  • Revert the changes to GM_xmlhttpRequest() in 0.9.13 (which were supposed to have no effect, but created a subtle bug). (#1472)
  • Guarantee that we do not create a specific kind of memory leak.  (#1482)

Thursday, November 03, 2011

Greasemonkey 0.9.13 Release

The entire list of bugs handled in this release is also available via the 0.9.13 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

Enhancements:
  • Script updates install automatically. (#1442)
    In 0.9.12 we added update checking.  In 0.9.13 updates install automatically, but there's still some discussion over how to handle this best (#1455), so checking is still not enabled by default.  We hope to figure this out by the 0.9.14 release.  Some other update related work was done, mostly to make Firefox 3 work better.  (#1421, #1422, #1423)
  • Provide an Options item in the Monkey Menu. (#1447)
  • Add a Cancel button to the Options window. (#1454)
Bug fixes:
  • GM_getResourceText() was broken. (#1438)
  • You can properly select an application (bundle) for the editor on Mac OS X.  (#1439)
  • Fix GM_openInTab() for Firefox 3. (#1444)
  • Fix when pages reassign the location during page load. (#1445)

Friday, October 28, 2011

Greasemonkey Usage by Firefox Version, Oct 2011

Just like August's post, here's a graph of what versions of Firefox people use Greasemonkey with.



You can open the spreadsheet for the raw data and a larger graph.

Takeaways seem to be not a lot has changed, besides some up-and-down for each rapid release version since then (6 came and went, with 7 replacing it).  Firefox 3.6 users are still just shy of three quarters of a million.  Firefox versions too old for the newest version of Greasemonkey are still hanging in at almost a quarter million, and just shy of 50,000 users are running a too-new version of Firefox, probably with Add-on Compatibility Reporter.

Friday, October 21, 2011

Greasemonkey 0.9.13 beta1

Greasemonkey release 0.9.13 should include complete automatic in-the-backgroudn updating of user scripts.  As yet there are no docs for script authors but the short version is: put an @version in your metadata, and increase it when there is an update.

Greasemonkey 0.9.13 beta1 is now available for testing.  It's in the Mozilla Add-ons beta channel. Open the "Development Channel" drop down at the bottom of the page, and click the "Add to Firefox" button.

Please report an issue or find us at the mailing list to provide feedback on beta versions.

Thursday, September 29, 2011

Greasemonkey 0.9.12 Release

The entire list of bugs handled in this release is also available via the 0.9.12 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to log an issue or let us know at greasemonkey-dev (and be clear that it's with this version).

Enhancements:
  • Script update checking (limited!). (#1053)
    We have started to build script update checking.  In 0.9.12, it is never automatic in the background.  You may choose to check one script (right click and choose "Find updates") or all scripts ("Check for updates" in the tool menu near the top of the Add-ons Manager).  If updates are found they will then be displayed.  Please use this feature and let us know if it works well, or if it breaks somehow.  Once we're confident that it works, we'll turn it on by default (automatically checking in the background) in a future release.
  • Scripts run with the latest available version of Javascript. (#1403)
Bug fixes:
  • Shared window object. (#1278)
    Before 0.9.0, it was possible to set properties on the window object, and other scripts would be able to read those properties.  This behavior was never supported, but as the compatibility fix that broke it is no longer necessary, it has been reverted.  It is still not suggested that you leverage this quirk, it is not a fully supported feature and may break again in the future.
  • Right-clicking a script in the Monkey Menu works, even when no editor is configured.  (#1405)
  • Fix launching the editor.  (#1409)
  • Scripts with @match could break loading all other scripts.  (#1414)
  • When browsing local files, menu commands for all scripts (on all tabs) no longer display.  (#1424)
  • Fix GM_openInTab(). (#1428)

Friday, September 23, 2011

Try the Greasemonkey 0.9.12 beta!

We've fixed a few bugs and started working on auto-updates for scripts. Please, if you have a chance, download the beta release and test things out.

It's available in the Mozilla Add-ons beta channel. Open the "Development Channel" drop down at the bottom of the page, and click the "Add to Firefox" button.

Please report an issue or find us at the mailing list to provide feedback on beta versions.

Tuesday, September 06, 2011

Greasemonkey 0.9.11 Release

The entire list of bugs handled in this release is also available via the 0.9.11 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to either let us know at greasemonkey-dev or log an issue (and be clear that it's with this version).

Bug fixes:
  • Work around for a problem Tab Mix Plus causes when opening links in new tabs. (#1406)

Monday, August 22, 2011

Greasemonkey 0.9.10 Release

The entire list of bugs handled in this release is also available via the 0.9.10 milestone on GitHub. Note that as always it takes some time for Mozilla to review the new version, but it's available on the all versions page immediately.

If you are using it and notice problems, it's best to either let us know at greasemonkey-dev or log an issue (and be clear that it's with this version).

Bug fixes:
  • You will be prompted to pick a new editor if the existing setting is an invalid path. (#1386)
  • Changing views within the Add-ons Manager could cause the user script sort order to be wrong. (#1394)
Enhancements:
  • When a script causes an unhandled error, the display in the Error Console will be a bit easier to read. (#1396)
  • When Growl is not installed, Greasemonkey's notifications are less obtrusive. (#1397)
  • A range of internal code clean ups. (#1395, #1398, #1399, #1400, #1401)