I ended up ordering similar LM3UU bearings and 3mm dowel rods from McMaster and although they're much more expensive, the tolerances are night and day compared to the generic LM3UU bearings received from amazon. If you, or others here, are looking to increase precision in the J5 assembly I think replacing the 4 bearings and 2 rods with parts from McMaster would be a good / worthwhile first step!
Bearings: https://www.mcmaster.com/61205K71
Rods: https://www.mcmaster.com/91585A376
Thanks again for all your work on this awesome project!