Rownd - Notice history

During our monitoring, we noticed a significant drop in traffic to some of our backend services.

Upon further investigation, we discovered that the internal TLS trust chain had broken due to the renewal of our private intermediate CA. This caused leaf certificates that depended on the previous intermediate CA cert to lose their trusted status, and so traffic to various internal services was affected. The result is that users may have experienced issues signing in and some platform accounts may have seemed like they were missing data.

Rest assured that no data was lost during this incident.

Moving forward, we will implement improved monitoring and processes that ensure the expiration of an internal root or intermediate CA is a) known well in advance, and b) automatically triggers the necessary processes that will refresh leaf certificates such that they tie back to the updated cert chain. This should prevent further incidents like this one.

We understand how important authentication is to every product, and we are committed to ensuring that our customers can depend on Rownd as a service with near 100% uptime.

We apologize for this incident and we will do better. If you have any further questions or concerns, please don't hesitate to reach out. We're happy to answer any inquiry.

Some of our load balancers failed to retrieve an updated certificate during our normal automated refresh cycle. We manually recycled the load balancing system and new certificates were retrieved successfully.

We are monitoring this situation to ensure that the system is not encountering other certificate-related issues and we are implementing processes to provide better visibility and awareness when such an issue arises in the future. We are also working on improvements to our infrastructure that will prevent this type of problem from occurring in the future.