[Resolved] Fonto & Chrome 80

Update 21st of February 2020

This issue has been resolved by the Chromium team. We verified that the problem is resolved in the following version:

  • Google Chrome - Version 80.0.3987.116 (Official Build) (64-bit)
  • Microsoft Edge - Version 80.0.361.57 (Official build) (64-bit)

Please make sure all your users update to the latest version and remove the temporary workaround from your environments it you installed it.

Postmortem and workarounds

During automated performance testing on Friday the 7th of February, one of the Fonto team members discovered quirky behavior in one of the editors we regularly exercise. In the specific case some portions of the loaded documents hierarchy.

After some analysis, we managed to track down weird behavior in one of the Array.reduce methods. Array.reduce is a common JavaScript function that is provided by all browsers. We could only reproduce the bug in Chrome 80 which was rolled out to users worldwide since the 4th of February. After further analysis, we managed to find an existing bug report in the Chromium bug tracker which confirmed our suspicion. This is when we sent out our communiqué to notify all our users.

One of our other team members managed to find a work-around for the bug. We needed more time to evaluate the stability and assess the impact on performance over the weekend while we were monitoring our support channels in case customers got blocked. More and more people reported the same problem and the Chromium team started to evaluate the impact of the bug.

On Monday we had more performance test data available to assess the impact. It turns out that the workaround as suggested by our team member had little observable impact. All but one test showed no regression and one test showed a 20% regression. At the same time the Chromium team was analysing the problem in detail: the issue was assigned the ReleaseBlock-Stable label indicating they considered this a blocking issue. In a subsequent comment showed that this was a candidate for the upcoming bugfix release of Chromium and Chrome.

Since none of our customers reported any issues, we decided not to add the workaround to the Fonto platform because it was likely that the bug would be fixed by the Chromium team relatively soon. We decided to write this postmortem which provides insight into the issue at hand and to provide the workaround as a patch that can be applied into production immediately if need be.

This page will regularly be updated with new information to the status of the bug and a potential bug fix rollout by Chrome. Our recommendation is to keep an eye on this page.

If you have any questions or if you may have observed this issue in the ‘wild’, please reach out to us via the regular support channels.

Affected browsers & operating systems

Based on the results reported in the Chromium bug report the following browsers versions are affected:

  • Google Chrome - Version 80.0.3987.87 (Official Build) (64-bit)
  • Microsoft Edge - Version 80.0.361.48 (Official build) (64-bit)
  • Google Chrome - Version 81.0.4033.2 (Dev) also known as M81
  • Google Chrome - Version 82.0.4054.0 (Canary) also known as M82

All operating systems running one of the aforementioned versions are affected.

This list may be out of date. Please consult the Chromium bug report for the latest information.

Affected versions of Fonto

All 7.x and possibly also 6.x versions of Fonto are affected by this bug. However, the bug required specific conditions to occur. So far the impact seems to be low: it may introduce visual glitches but so far we have not seen any cases where actual work was lost.

Visual glitches that we are currently aware of are:

  • Parts of the document hierarchy can not be seen in the content view nor in the outline view when using single-document-hierarchy. The invisible content will still be saved, it is just not accessible to the author.
  • We have seen widgets in the content view to be displayed twice instead of once.
  • We have seen a custom hierarchy omit parts of the hierarchy when using XPaths with more than one step to build them.

This bug does not only affect Fonto; the bug may also occur in other web applications, depending on the JavaScript used there. This may include your CMS.

Temporary workarounds

In order to mitigate any risks, there are two workarounds available:

  1. Use an unaffected browser; Firefox or Safari are not affected by this bug.
  2. Install a temporary code change; See the procedure below.
Both workarounds are temporary. As soon as the affected browsers are fixed, the workarounds should be removed. We’ll update this page regularly as soon as new information is available.

Installing the temporary code change

This workaround can be applied without updating the Fonto platform. This minimizes the impact and can be applied quickly on dev, test, acceptance and/or production.

This fix is applied on the Fonto build. With every rebuild of the Fonto configuration, this workaround needs to be reapplied manually.

Procedure:

  1. Open index.html of your build in a text/code editor.
  2. Paste the following snippet in the <head> of index.html, at the very bottom:

    <script type="text/javascript" src="./chrome-80-workaround.js"></script>
  3. Save your changes.
  4. Download chrome-80-workaround.js.
  5. Place it in the same folder as index.html.
  6. Deploy the new and modified files to the relevant servers (dev, test, acceptance and/or production).

We recommend performing a regression test to ensure any observed issues are no longer present and no new issues are introduced.

Conclusions

We’re continuously monitoring the situation. For now, Fonto will not issue a hotfix for this specific bug since it is likely to be fixed within the browsers in the next few days. We provided two workarounds in case you’re running into production issues.

We’ll regularly update this page with new information as soon as it comes available. Please keep an eye on this page. If you have any remaining questions, please reach out to us via the regular support channels.