There are two different sets of configuration files:
On the server but for the user.
HOME/.ssh/config
See SSH - OpenSSH Server (sshd)