On Monday, June 20th, 2022, the Recorded Future risk scoring feature was affected by a bug introduced during a platform update that resulted in incorrect calculation of risk scores for many entities, across all entity types. As a result, false positives due to inflated suspicious and malicious risk scores on IOCs (indicators of compromise) were experienced across all parts of the platform including the Portal, Alerts, API, and our partner integrations.
The effects of this issue causing inflated risk scores and resulting false positives were encountered in a variety of ways:
Although this incident affected all types of indicators in the platform, the total number of entities affected made up roughly 5% or less of each of our default risk lists for each entity type.
As part of our product release during the morning of June 20th, there was an update to an external library used by our risk scoring code. The update in question included a breaking change that wasn’t highlighted in the library’s change log. As a result, risk rules were computed incorrectly.
The Recorded Future data science and platform engineering teams kicked off several processes to recover from this incident including both a manual correction effort to remediate individual entities, largely domains, that were reported as high-impact false positives and an automated process to reevaluate risk rules for all impacted entities. Our initial focus was on reprocessing and updating the risk scores related to entities that were inaccurately put on our default risk lists because of the error.
Monday, June 20th - Issue first reported in the morning EDT. By 12pm EDT, it was determined to be a systemic issue related to risk scoring. Manual fixes of individual, incorrect scores began.
Tuesday, June 21st - Early morning EDT, new false positives were stopped. The root cause was confirmed. Shortly after a fix was in place and systemic reprocessing of risk scores began.
Wednesday, June 22nd - Reprocessing of risk scores continued with priority given to domains incorrectly scored > 65 (malicious/very malicious level) to reduce further impact on integrations.
Friday, June 24th - By 2pm EDT, all domain risk scores and most IP and URL risk scores were corrected. Risk scores related to affected hashes and other entities were still being reprocessed.
Monday, June 27th - By afternoon EDT, all IPs and URL indicators with scores of 65 or higher had been fixed. This is in addition to domains with scores 65+, which were fixed previously.
We are taking several actions as priority to prevent and mitigate impact of similar systemic issues related to risk scoring in the future. This includes steps to better avoid, identify, verify, revert, and recover more efficiently.