[Serverless] [GCP] [us-central1] High upsert latency for the 90 percentile

Incident Report for Pinecone

Resolved

This incident has been resolved.
Posted Dec 18, 2024 - 21:14 UTC

Update

We are continuing to monitor for any further issues.
Posted Dec 18, 2024 - 15:32 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Dec 18, 2024 - 15:32 UTC

Update

As part of mitigation for the elevated upsert latency, we are performing a few minutes of maintenance to scale up database resources. We anticipate a drop in write path availability.
Posted Dec 18, 2024 - 15:12 UTC

Investigating

We are investigating an issue where one in a hundred upsert operations takes over a second. This only affects serverless indexes in GCP us-central1.
Posted Dec 17, 2024 - 07:20 UTC
This incident affected: Serverless Indexes (GCP us-central1).