concurrency: add metrics for virtual intent resolution#164641
Open
stevendanna wants to merge 3 commits intocockroachdb:masterfrom
Open
concurrency: add metrics for virtual intent resolution#164641stevendanna wants to merge 3 commits intocockroachdb:masterfrom
stevendanna wants to merge 3 commits intocockroachdb:masterfrom
Conversation
…quests If we encounter a LockConflictError after our first attempt to virtually resolve all of our intents, it must mean that we hit the maximum lock conflicts on our first scan. In this case, we want to avoid a situation where we end up with either (1) a huge number of lock updates in memory, or (2) a livelock in which we the lock table memory limit continues to evict previously found locks, resulting in a virtual resolve set that is never complete. We solve this by condensing all virtually resolved locks to per-transaction resolve range requests after the first retry. If we hit too many resolve range requests, we then give up on virtual intent resolution. Fixes cockroachdb#163924 Release note: None
We recently added PrepareForLockConflictRetry to the concurrency manager which, for VIR-enabled requests, condenses the previous set of LockUpdates into a smaller set of ranged LockUpdates. This solved a situation in which upon re-entering ScanAndEnqueue after a LockConflictError, we did not depend on those same locks being in the lock table. This, however, overlooked another, perhaps even more common, case: - Request enters ScanAndEnqueue - Request encounters some number of locks and pushes their holders - Other requests push at least one of our updated transaction statuses out of txnStatusCache. - Request re-enters ScanAndEnqueue, encounters the same locks, but now must re-push it. This commit introduces a request-scoped map of transaction statuses to ensure we never have to re-push a request we previously pushed. Along the way, it refactors our handling of toResolve condensing. We are also considering other alternatives: - Using a much larger txnStatusCache to avoid this issue in practice most of the time. - Preserving toResolve across calls of ScanAndEnqueue for VIR-enabled requests. This is a bit tricky to do efficiently since without care it would result in duplicates in toResolve or excessive allocations to maintain our set. Epic: CRDB-49120 Release note: none
Contributor
|
Merging to
|
|
Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Member
|
This change is |
Add four metrics to provide observability into VIR behavior: - kv.concurrency.virtual_resolve.condense: number of times point intent resolutions were condensed into range resolutions during scanning. - kv.concurrency.virtual_resolve.disabled: number of times VIR was disabled for a request due to excessive range resolve accumulation. - kv.concurrency.virtual_resolve.intent: number of point intents resolved virtually during read evaluation. - kv.concurrency.virtual_resolve.intent_range: number of range intent resolutions resolved virtually during read evaluation. Epic: CRDB-42978 Release note: None
stevendanna
force-pushed
the
ssd/vir-metrics
branch
from
537fa29 to
43d840b
Compare
stevendanna
requested review from
herkolategan and
nameisbhaskar
and removed request for
a team
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add four metrics to provide observability into VIR behavior:
resolutions were condensed into range resolutions during scanning.
disabled for a request due to excessive range resolve accumulation.
resolved virtually during read evaluation.
resolutions resolved virtually during read evaluation.
Epic: CRDB-42978
Release note: None