Unable to authenticate local user

We encountered one other weird problem today when someone reported that they cannot login as local user and also su to that user doesn't work as normal user.

Error were as follows:

#1.

$su - pankaj

su: incorrect password

#2.

users listed in /etc/passwd (local user) cannot login to the server

Logs:

/var/log/message shows

Oct 29 14:57:15 pgserver01 sshd[1457]: Address 11.22.20.130 maps to l4339284.federated.fds, but this does not map back to the address - POSSIBLE BREAKIN ATTEMPT!

Oct 29 14:57:15 pgserver01 sshd(pam_unix)[1465]: auth could not identify password for [pankaj]

Oct 29 14:57:15 pgserver01 sshd[1457]: error: PAM: Authentication failure for pankaj from 11.22.20.130

Oct 29 14:57:17 pgserver01 sshd(pam_unix)[1457]: auth could not identify password for [pankaj]

Oct 29 14:57:17 pgserver01 sshd[1457]: Failed password for pankaj from ::ffff:11.22.20.130 port 56138 ssh2

We did lot of troubleshooting around what is the exact symptoms and restarted all necessary services to clear out any hand auth modules.
few steps taken:

/etc/init.d/vas restart

/etc/init.d/xinetd restart

/etc/init.d/sshd restart

Resolution:

We found misconfigued the pam.d/system-auth with option "use_first_pass"

[root@esu1l101 ~]# cat /etc/pam.d/system-auth

#%PAM-1.0

# This file is auto-generated.

# User changes will be destroyed the next time authconfig is run.

auth required /lib/security/$ISA/pam_env.so

auth [ignore=ignore success=done default=die] pam_vas3.so create_homedir

#auth sufficient /lib/security/$ISA/pam_unix.so likeauth nullok use_first_pass <-- replaced this line with line below

auth sufficient /lib/security/$ISA/pam_unix.so likeauth nullok

auth required /lib/security/$ISA/pam_deny.so

account [ignore=ignore success=done default=die] pam_vas3.so

account required /lib/security/$ISA/pam_unix.so

account sufficient /lib/security/$ISA/pam_succeed_if.so uid <>

account required /lib/security/$ISA/pam_permit.so

password [ignore=ignore success=done default=die] pam_vas3.so

password requisite /lib/security/$ISA/pam_cracklib.so retry=3

password sufficient /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow nis

password required /lib/security/$ISA/pam_deny.so

session required /lib/security/$ISA/pam_limits.so

session required pam_vas3.so create_homedir

session required /lib/security/$ISA/pam_unix.so

PAM optional arguments module explaination:

use_first_pass

The module should not prompt the user for a password. Instead, it should obtain the previously typed password

(from the preceding auth module), and use that.

If that doesn't work, then the user will not be authenticated.

(This option is intended for auth and password modules only).