Dear Mergin Maps user,
We’re sorry for the service outage that occurred yesterday (6th February 2022). The good news is that all your data are safe and the system was not damaged, breached or compromised in any way. The interruption was due to unforeseen issues during a major upgrade to Workspaces (https://merginmaps.com/blog/introducing-workspaces-simplified-collaboration). The issue may have also caused one or more emails (see below) to be sent to you by mistake. Please ignore such emails.
Let me explain the issue in more detail: A scheduled service update for the deployment of the new workspaces feature took place yesterday between 12:00-13:30 CET. The deployment appeared to be successful in post-deployment testing so the service was taken out of maintenance mode. At 14:00 CET, many of you received emails (e.g. the end of the trial or similar) that were not valid for your user account. Around 14:30 CET an unforeseen database issue caused the service to become unavailable. We identified the problem shortly after it manifested itself and then continuously worked to devise and safely apply a set of fixes with the service coming back online around 22:00 CET. The web map component and part of the email-sending component remain disabled for the time being but should soon be back working again.
For those who are interested in the technical detail, the problem was identified as a rare case when the database query planner makes a less than optimal choice of how to execute a particular SQL statement (containing SELECT ... ORDER BY joined_attribute LIMIT 1). A query that normally ran well under a second ended up taking more than 80 seconds, rapidly exhausting database resources and effectively making the service unavailable.
Again, we would like to apologise for any inconvenience this service outage may have caused. It’s unfortunate that even with multiple rounds of testing prior to release and various existing safety measures, we were not able to catch this issue. We are working to get the suspended features back as soon as possible and will be using the experience to continuously improve our procedures, further mitigating risks of such events in the future.
Thanks to all the users who got in touch with us to report issues. If you still have any problems, please do not hesitate to contact support@merginmaps.com and we will get back to you promptly.
On behalf of the whole Mergin Maps team,
Peter Petrik and Saber Razmjooei