Don't Like the Cutpoints? Call Jenny.

One way I like to drive myself insane is trying to recreate CMS cutpoints from released final data. Of course, CMS acknowledges the futility of such efforts on p24 of the tech specs. That doesn’t mean it’s not a good idea to try though, as you can learn a lot about the measures and how they might move in the future.

One key insight is the algorithmic uncertainty that plagues the methodology. There actually is no guaranteed convergence of the algorithm described as “hierarchical clustering with 10-group mean resampling.” While mean resampling helps, it does not completely stabilize the estimates. A cute random seed is chosen to ensure exact reproducibility. It’s important to remember though that this seed is for computational reproducibility, which is very different from algorithmic convergence. One only need run the same unseeded algorithm on the same data multiple times to show this is the case—whether you use R, SAS, or Python can change some minor technical details but not the underlying fact that cutpoints can vary even in the absence of industry growth due just to the algorithm itself.

For some measures this matters more than others. I see very little algorithmic variance in some measures but sometimes the difference can be 4-5 percentage points. This means that many plans’ star ratings on measures are determined by the 8675309 lottery ticket (the random seed used).

Of course, the other major obstacle to recreating cutpoints is missing data from consolidations and terminations—essentially data CMS used in the clustering but is not published in final. This is where it can get interesting. Note in this example how the 2-star for Cholesterol adherence on the published data is actually 81 or 82 via the algorithm. The censored data could be the reason this cutpoint is pulled up considerably. Plans got algorithmically unlucky with the 3-star cutpoint as there is much more mass on 87 (though these probabilities would change as well with the censored data published so how unlucky that draw was is hard to say).

All this really highlights some algorithmic issues with CMS’ current approach. It’s no wonder then that they are considering a switch to something like quantiles. I think there are some other candidate approaches as well. I’m not sure plans are paying enough attention to this potential rule change but it’s certainly something you want to get ahead of and have a position on before it’s too late.