It took me about a year to redesign our system recently after this was discovered during a health check.
Each site takes up quite a bit of RAM, so they should be limited.
Asset Groups should be used where to need to assign things to or filter Site/Scan Schedule membership.
Tag filters are the best way to populate enrichment data, then filtering on these Tags, just like you are doing with Tagging your location then having a Asset Group look for those Tags. I originally had a Static Asset Group to do the Tagging and I was told this is doubling the effort and from ICON it is easier to dynamically Tag vs update Group Membership.
Take a look at this thread. Also, ask your rep about a health check.