Setting up Federated Search Between Two Splunk Enterprise Deployments: with Enterprise Security 

Bringing all that data together for a unified view can be difficult when organizations run multiple Splunk environments, such as one supporting modern security operations and another handling legacy logs. We solved this by setting up Federated Search between our Splunk Enterprise deployments with Enterprise security (ES) enabled. 

Here are the steps in detail. 

Environment Details 

• Splunk Versions: 
  • Provider: 9.1.7 
  • Local: 9.2.4 
• Enterprise Security: 7.3.0  

Federated Search: Step-by-Step (Transparent Mode) 

On the remote Splunk (provider): 

• Create Role and User: 
  • In Splunk Web, go to Settings > Roles and set up a new role, giving it fsh_manage and search capabilities. Choose which indexes to expose rather than inheriting from admin or power roles. 
  • Then, make a service user with that role, setting up a default password and time zone. 
• Deselect require password change on first login and save.  

On the local Splunk (consumer/searcher): 

• Configure the federated Provider: 
  • Under Settings > Federated Search, add a new provider. 
  • Choose “Splunk” and “Transparent Mode.” Add in the provider’s hostname/port and service account credentials. 
  • Evaluate the connection. if it passes, save. 

Compatibility with Splunk Enterprise Security 

• If Splunk ES is installed on the provider's search head, Federated Search supports it. 

Additional Notes: 

• Transparent mode means the real data stays remote; you just fetch the results, not the underlying data set. 
• Avoid mixing Transparent and Standard modes in a single deployment. 
• SSL is advised for security. 
• Monitor for latency in federated searches, especially with large datasets or distant deployments.

Pros and Cons 

Pros 

• Unified searching across different Splunk instances. 
• No extra data storage costs as data are not duplicated. 
• Ensures compliance by leaving data in its original location. 
• Enables instant access to remote data. 
• Ideal for distributed environments. 
• As the environment grows, this makes it possible to add more sites or providers. 

Cons 

• Complex queries can be slower, especially over distance. 
• You will need reliable network links. 
• Certain sophisticated transformations and SPL commands cannot be executed remotely are not repeated. 
• Extra configuration effort and a bit more to troubleshoot. 
• Security and licensing need careful review. 

Limitations 

• Do not install ES on remote search heads when using transparent mode. 
• Commands like meventcollect, rest, and from are unsupported.
• Accelerated data models need to be referred locally. 

Monitoring and Troubleshooting 

To identify the issues with federated search setup, the following query can be used to set up alerting. 

index=_internal component=federated* log_level IN (ERROR, WARN) 

Summary: 

Federated Search supports transparent mode with Splunk ES, recognizing limitations.  

For more information, explore our Splunk Platform support & Maintenance 

References: 

• Federated Search in Transparent Mode (ES) 
• Standard vs. Transparent Modes 
• Running Searches Over Transparent Mode