diff --git a/Benchmarking-your-changes.md b/Benchmarking-your-changes.md index ffe95dc..d7158c8 100644 --- a/Benchmarking-your-changes.md +++ b/Benchmarking-your-changes.md @@ -1,3 +1,8 @@ +Two options for benchmarking: +1) [Using the Test Runner](#using-the-test-runner) +2) [Using Firefox Talos](#using-firefox-talos) + +## Using the Test Runner When working on issues related to performance, it is important to provide a performance benchmark for your changes to assess whether or not your change has a performance impact. PDF.js provides tools to do this easily. Normally you would create a simple manifest file has a couple of PDF files you're trying to optimize and run it multiple times, e.g. `benchmark.json`: ``` @@ -34,4 +39,50 @@ Finally, you can compare the measurements and see any performance differences: As a sanity check, you should do this twice with the same code and compare the results. -For faster benchmarking during development, it could be useful to pass the `--noChrome` command line parameter to `test.js` to skip Chrome and only test with Firefox. \ No newline at end of file +For faster benchmarking during development, it could be useful to pass the `--noChrome` command line parameter to `test.js` to skip Chrome and only test with Firefox. + + +## Using Firefox Talos + +### About +Versions of PDF.js that are merged in to Firefox are automatically benchmarked as part of the talos tests to monitor performance regressions/improvements. The talos [pdfpaint](https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#pdfpaint) test benchmarks how long it takes from a PDF being opened to the first page of the PDF being rendered. The advantage of using this system for benchmarking is that it is closer to a real world benchmark where we test the most important thing, which is how long it takes for the first page to render. + +### Running Locally + +#### Prerequisites +This assumes you've setup a [local PDF.js repo](https://github.com/mozilla/pdf.js#getting-the-code) and you've setup a [build of Firefox locally](https://firefox-source-docs.mozilla.org/setup/index.html). We recommend using an non-debug artifact build of Firefox. + +#### Adding a Custom PDF +_If you just want to benchmark your changes with the default PDFs you can skip this step._ +1. Copy the PDF file into `/testing/talos/talos/tests/pdfpaint/` +2. Add the file to the manifest in `/testing/talos/talos/tests/pdfpaint/pdfpaint.manifest` +3. (Optional) Remove any other PDF files in the manifest if you only want to benchmark your PDF + +#### Record the Baseline Data +First, we want to run the test PDF without any changes to see how it will compare. In the `` run: + +`./mach talos-test -a pdfpaint` + +When the test finishes there will be a section in the terminal with `------- Summary: start -------` and `-------- Summary: end --------`. You'll want to copy that section down somewhere for comparison. + +#### Apply Your Changes & Test Again +1. In the ``: + 1. make your changes + 2. build `gulp mozcentral` + 3. copy the pdf.js build to firefox `rsync -a -v /build/mozcentral/browser/extensions/pdfjs/ toolkit/components/pdfjs/` +2. Back in ``: + 1. Build firefox `./mach build faster` + 1. Run the tests again `./mach talos-test -a pdfpaint` + +The talos tests will run and output a test summary. You can now compare these results to see how your changes affected performance. + +### Running pdfpaint on Mozilla's infrastructure +Similar to above, you'll want to create a baseline result without your changes and another result with your changes. The pdfpaint test is part of the talos "other" group. An example of running the tests 6 times on windows: + +`./mach try fuzzy -q 'windows10-64-shippable opt-talos-other' --rebuild 6 --artifact` + +Once the tasks finish, you can then compare the results by using the two commit IDs and replacing `` and `` below: + +``` +https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=&newProject=try&newRevision=&framework=1&showOnlyImportant=1&page=1 +``` \ No newline at end of file