Load balancing with SAML2

I want to protect an application which is load balanced. So I have multiple instance of the application and then multiple instances of the SAML Service Provider.

From the external, only 1 URL is known and its the load balancer job to tell to which server it will be sent.

If the SP redirect the user to an IDP with an authnrequest, the IDP will send the response to the SP URL (the same for all SPs), but the load balancer can decide to send this response to any SP available.

So a SP can receive a response intended for another one....

Does anybody have already think about this kind of problem ?

Valerie