Confidential Input Hurts the Marginalized