On December 8th at 11:26 AM ET, Double experienced an issue that caused all incomplete closing tasks to be marked as canceled.

This affected every practice immediately, and any tasks created over the next 1 hour and 20 minutes were also impacted. The underlying issue was remediated at 12:26 PM ET, stopping additional tasks from being marked canceled. Throughout the rest of the day, we restored all affected tasks to their prior statuses, completing the process at 8:39 PM ET.

Below is a post-mortem from our CTO Kenny explaining what happened and the steps we are taking to prevent issues like this from happening in the future.

I am profoundly sorry for the disruption we caused your practices yesterday. We take our responsibilities seriously, and we know how critical it is for you to be able to rely on your workflow systems. We will work every day to earn back your trust.

As a small gesture of our apology, we have issued an automatic credit for the cost of yesterday’s subscription. You should expect to see applied to your account in the coming days. You should not have to pay for software you can’t use.

If you are still experiencing any lingering issues or have any further questions, please contact our support team at help@doublehq.com, and we will make sure they are addressed with the highest priority.

Post-mortem on December 8 Task Issue

At 11:26 AM ET, we deployed a code change to fix a minor bug with archived clients. However, this code accidentally affected all closing tasks that were not in the statuses done or canceled. Furthermore, any task created from 11:26AM ET time through 12:26PM ET time was automatically marked as canceled.

After an initial investigation, we identified the issue and reverted the code change at 12:26 PM ET.

For remediation of the affected closing tasks, we leveraged our backup database and activity logs to restore each closing task to its original status.

For any closing tasks created between 11:26AM ET time and 12:26PM ET time, we’ve updated those tasks to “Not Started”.

These updates were processed continuously over the course of the day and completed by 8:39 PM ET. No custom tasks or tax tasks were affected.

We have put multiple measures in place to prevent this from ever happening again. Those measures include additional QA resources, enhanced auto-locking mechanisms to prevent tasks from being updated at one time if conditions aren’t met, and enhanced unit+integration testing with our backend implementation. We will also continue to make investments with our activity log so that your team has even more observability of your data.

• 11:26 AM ET - code change pushed. All incomplete closing tasks marked as canceled.
• 12:26 PM ET - code change reverted and data restoration process commenced.
• 2:22 PM ET - data restoration script initiated.
• 8:39 PM ET - data restoration script completed.