Are there org level constraints around creating custom roles?

No.

Are there org level constraints around granting Cross Account access to service accounts?

Yes. There is a constraint that will limit the scope of  a service account to it's own project..

Should you ever create custom roles?

Bucket creation without deletion. Storage Admin Role as an example of why custom roles are needed

Should you ever grant IAM.setIamPolicy for any user or service account?

The ability to set IAM policies is essentially the ability to grant IAM access to users or service accounts.

These two roles are to be used with a lot of precaution - they are in effect no different from granting the primitive project owner or folder owner roles.

roles/resourcemanager.projectIamAdmin

roles/resourcemanager.folderIamAdmin

Will allow you to add the 8000 plus permissions to create your own custom role EQUAL to a PROJECT OWNER

How do I mitigate the excessive grants present as part of the default GCP Service Accounts?

option 1 - Create Custom SA and disallow key creation on SA

option 2 - Limit Scope on VM (resource) Creation Time

What roles should I expect to see on a project level? On a SA level? - Service Account Specific Roles

On a SA level, you should see ServiceAccountUser and the token creator roles - that's it.

On a project level, you can see Service account creator, but should never really have the SA User and Token Creator on the project level. Often, the SA user is granted at the project level, making it possible for the SA User to use ANY service account in that project.