I wonder if you ran into a “false positive” sort of situation.
Your early/initial setup(s) did not appear to be wildly off, visually at least.
Looking back at my notes from the 14b sf I did, started at .02, it was decent, went to .03 was clearly deep, went to .025 was still clearly deep, went all the way down to .015, was clearly shallow. I can recall exactly, but think ended up back at .02 as the final setup that provided the best pattern. I’ll dig up the pictures showing that.
Point I’m getting at here is that on that setup within ±.005 of the final setup showed clear shallow and deep patterns, so .01, whereas you have gone way farther than that and it really hasn’t gone clear shallow or deep if that makes sense.
Which is for me where the confusion begins. But what I’m not sure of, is if the relationship between the diameter and tooth counts of the ring and pinion make any significant impact on the “slope” or rate of change. For example does a 3.07 gear sets pattern move “faster” or slower with a .003 change in pinion depth than a 4.88, assuming everything else is good and backlash held as a constant. Or do is there no difference at all or one that’s small enough to be a non factor.
Just more thinking out loud than anything here.