Is there a tool / recommended process (bb-config.properties flag, bb-manifest.xml tag, gradle build flag, etc) for searching out any non-public APIs in use in a custom B2?
I do not know of a tool for this, but hope that Scott Hurrey can reply since I am also interested in a way to do this besides manually going through custom B2 code and checking all method calls against Bb APIs corresponding to release used.
This is what I'm doing at the moment, and had the big shock that the DAO framework isn't in the Public API yet (at time of 3500.0.0's release).
You could look at the method signatures of various class files, which I know is monotonous and not particularly illuminating. The fact that the non-public API's are 'not public' suggests that there won't be much in terms of documentation (and I believe their use is 'discouraged' as they are considered as 'not stable' for public consumption, but not like that will stop anyone). And depending on the tooling used, that could be regarded as reverse engineering, which is also not allowed.
Assuming the Custom B2 is for Q2 2018 and uses only public APIs (which does not seem to be the case here), I think you could look at the 'View API Changes' API Differences between Building Blocks API version 3400.0.0 and Building Blocks API version 3500.0.0 to check still using only public APIs before rebuilding for Q4 2018.
Retrieving data ...