Unscheduled maintenance required for Monk Donate / Checkout
Incident Report for MonkDev
Postmortem

On April 15, 2015 a new server was created for Monk Donate and Monk Checkout in order to meet new PCI compliance standards. While the new server was being configured, it erroneously began running subscription payments alongside the current server.

This situation continued for 13 days while final preparations for the server migration were made. Unfortunately no information was found on our end to indicate a problem, and no support cases were received, although some donors noted that their payments had failed. As a result, some users saw some subscription payments charged twice, or saw an error stating, "A duplicate transaction has been submitted" inside the "Subscription Payment History" screen in Monk Checkout.

After receiving support cases noting the issue on April 28, we quickly disabled both the Monk Donate and Monk Checkout systems while the subscriptions database was reconfigured. After several days of rigorous testing, we were confident that payments could run smoothly, and both systems were up again on Tuesday, May 5. As a result, all subscription payments that were halted during April 28 - May 4 were all run on Tuesday May 5.

In short, there are two takeaways we wish to highlight:

1) ISSUING REFUNDS

During the week of April 27 - May 1, the administrators of all Monk Donate accounts that contained a user that was double-charged were sent an email including instructions for refunding the end user for the duplicate transaction. If you are a Monk Donate administrator and have any questions about these transactions, please contact us at support@monkdevelopment.com or by calling (877) 452-0015 x2.

2) CATCH-UP TRANSACTIONS

Though many recurring subscription payments were run on May 5 to make up for the resulting downtime, donors will continue to have their future transactions run on the day that they originally chose.

Again, if you have any questions or concerns please contact us at support@monkdevelopment.com or by calling (877) 452-0015 x2.

Thanks very much!

Posted May 07, 2015 - 15:00 PDT

Resolved
This incident has been resolved.
Posted May 07, 2015 - 14:50 PDT
Monitoring
MonkDev engineers have finished a set of updates to bring Monk Donate/ Monk Checkout back online and the team is monitoring the system closely as we approach the full resolution of this issue. Thank you!
Posted Apr 29, 2015 - 13:41 PDT
Update
Our team has identified the cause of the issue that arised yesterday in which some donors' transactions either generated multiple email notifications or were actually run twice in error. We will update you with more information as soon as possible. Thanks for your patience and support.
Posted Apr 28, 2015 - 15:33 PDT
Identified
Monk Donate / Checkout will be down for maintenance for a short period today as the team integrates some improvements to the system.
Posted Apr 28, 2015 - 10:14 PDT