... I suspect that if a solvent existed it would be extremely dangerous and not available to me anyway.
There is a silicone remover product available in retail. It is specific to silicone and is not as effective on other types of adhesives or caulking compounds.
We have used the DAP product.
DAP Silicone Sealant Remover Apparently
WD-40 can be applied to good effect. I use WD-40 for general goop removal but not (yet) for silicone.
Google search turned up these products which I had not seen before.
DSR-5 McKanicaź Silicone Caulk Remover Gel (0354) I use a
Razor scraper to shave the surface as much as possible. I find the
single edged razor blade scraper to allow a very close 'shave' of the surface, which often removes all but the thinnest residual film of unwanted material.