‘Major policy differences’ between Social Democrats and larger parties suggest Soc Dems unlikely to have role in next coalition