Routine database maintenance led to unexpected performance degradation
Cause:
A new node was added to the database cluster used for verification of application logins. The process of adding the node used was the same documented process followed for adding nodes to all other database clusters, which is routinely performed (several times per week).
It appears that we encountered a rare bug with the database cluster software which caused the new node to be treated as a primary node even though it was added as a backup node, which caused massive latency for the cluster, even blocking reconfiguration commands.
Remediation / Path Forward:
We have updated our node addition process to include even more safety measures in order to reduce the risk of similar future occurrences.
We have contacted the database cluster software vendor to notify them of this bug.
Posted Feb 16, 2025 - 20:38 EST
Resolved
Dear Client,
The Recorded Future application Portal, API, alerting, and mobile application services have now been restored. Our teams are actively working to monitor the performance of the platform, and a Root Cause Analysis (RCA) will be posted here once complete.
Regards, Recorded Future Platform Operations
Posted Feb 07, 2025 - 08:39 EST
Update
Dear Client,
We are currently experiencing a service disruption which affects the Recorded Future application Portal, API, alerting, and the mobile application. Our product operations team are actively working on identifying the issue and restoring accessibility to an optimal level of service
We will continue to provide updates as they become available. Please contact our support team at support@recordedfuture.com if you have any questions.
Regards, Recorded Future Platform Operations
Posted Feb 07, 2025 - 08:04 EST
Investigating
Access to the Recorded Future application and API is currently impaired, and we are currently investigating.
Posted Feb 07, 2025 - 07:53 EST
This incident affected: User Interface, Alerts, API, and Mobile app.