Add the directory only to the build interface, not to the install interface. INTERFACE_LINK_DIRECTORIES can't use paths that are prefixed in the source (or build) directory. Closes #22524.