`
sjk2013
  • 浏览: 2183464 次
文章分类
社区版块
存档分类
最新评论

Oracle 11gR2 RAC TNS-12542: TNS:address already in use 故障一例

 
阅读更多


一．问题描述

一朋友在生产库的一个节点上做了一个非常危险的操作，然后该节点的数据库监听出现了异常。

数据库是2个节点的RAC，版本11.2.0.3。

[grid@dave-db1trace]$lsnrctlstatusLISTENER

LSNRCTLforLinux:Version11.2.0.3.0-Productionon22-FEB-201318:40:27

Copyright(c)1991,2011,Oracle.Allrightsreserved.

Connectingto(DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUSoftheLISTENER
------------------------
AliasLISTENER
VersionTNSLSNRforLinux:Version11.2.0.3.0-Production
StartDate22-FEB-201318:00:03
Uptime0days0hr.40min.23sec
TraceLeveloff
SecurityON:LocalOSAuthentication
SNMPOFF
ListenerParameterFile/oracle/11.2.0.3/grid/network/admin/listener.ora
ListenerLogFile/oracle/app/grid/diag/tnslsnr/dave-db1/listener/alert/log.xml
ListeningEndpointsSummary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
Thelistenersupportsnoservices
Thecommandcompletedsuccessfully

此处显示：Thelistenersupportsnoservices

[grid@dave-db1trace]$lsnrctlstatusLISTENER_SCAN1

LSNRCTLforLinux:Version11.2.0.3.0-Productionon22-FEB-201318:40:46

Copyright(c)1991,2011,Oracle.Allrightsreserved.

Connectingto(DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUSoftheLISTENER
------------------------
AliasLISTENER_SCAN1
VersionTNSLSNRforLinux:Version11.2.0.3.0-Production
StartDate22-FEB-201317:48:19
Uptime0days0hr.52min.27sec
TraceLeveloff
SecurityON:LocalOSAuthentication
SNMPOFF
ListenerParameterFile/oracle/11.2.0.3/grid/network/admin/listener.ora
ListenerLogFile/oracle/11.2.0.3/grid/log/diag/tnslsnr/dave-db1/listener_scan1/alert/log.xml
ListeningEndpointsSummary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Thelistenersupportsnoservices
Thecommandcompletedsuccessfully

手工注册后，问题依旧：

SQL>alter system register;

节点2上的监听是正常的：

[oracle@dave-db2admin]$lsnrctlstatus

LSNRCTLforLinux:Version11.2.0.3.0-Productionon22-FEB-201318:38:06

Copyright(c)1991,2011,Oracle.Allrightsreserved.

Connectingto(ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUSoftheLISTENER
------------------------
AliasLISTENER
VersionTNSLSNRforLinux:Version11.2.0.3.0-Production
StartDate22-FEB-201317:30:45
Uptime0days1hr.7min.21sec
TraceLeveloff
SecurityON:LocalOSAuthentication
SNMPOFF
ListenerParameterFile/oracle/11.2.0.3/grid/network/admin/listener.ora
ListenerLogFile/oracle/app/grid/diag/tnslsnr/dave-db2/listener/alert/log.xml
ListeningEndpointsSummary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.106.14)(PORT=1521)))
ServicesSummary...
Service"+ASM"has1instance(s).
Instance"+ASM1",statusREADY,has1handler(s)forthisservice...
Service"DAVE"has1instance(s).
Instance"dave2",statusREADY,has1handler(s)forthisservice...
Service"oradb"has1instance(s).
Instance"dave2",statusREADY,has1handler(s)forthisservice...
Service"oradbXDB"has1instance(s).
Instance"dave2",statusREADY,has1handler(s)forthisservice...
Thecommandcompletedsuccessfully

--使用PS 查看监听的进程：

[grid@dave-db1trace]$ps-ef|grepLISTENER
grid4211017:48?00:00:00/oracle/11.2.0.3/grid/bin/tnslsnrLISTENER_SCAN1-inherit
grid14351018:00?00:00:00/oracle/11.2.0.3/grid/bin/tnslsnrLISTENER-inherit
grid44821333018:39pts/100:00:00grepLISTENER
grid14380102012?00:02:56/oracle/11.2.0.3/grid/bin/tnslsnrLISTENER-inherit
grid15075102012?00:03:07/oracle/11.2.0.3/grid/bin/tnslsnrLISTENER_SCAN1–inherit

注意1：

这里是有问题的，因为监听的进程出现了2次。 我们继续看。

[grid@dave-db2admin]$crsctlstatresource-t
--------------------------------------------------------------------------------
NAMETARGETSTATESERVERSTATE_DETAILS
--------------------------------------------------------------------------------
LocalResources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
ora.LISTENER.lsnr
ONLINEINTERMEDIATEdave-db1NotAllEndpointsRegistered
ONLINEONLINEdave-db2
ora.asm
ONLINEONLINEdave-db1Started
ONLINEONLINEdave-db2Started
ora.gsd
OFFLINEOFFLINEdave-db1
OFFLINEOFFLINEdave-db2
ora.net1.network
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
ora.ons
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
ora.registry.acfs
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
--------------------------------------------------------------------------------
ClusterResources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1ONLINEINTERMEDIATEdave-db1NotAllEndpointsRegistered
ora.cvu
1ONLINEONLINEdave-db1
ora.dave-db1.vip
1ONLINEONLINEdave-db1
ora.dave-db2.vip
1ONLINEONLINEdave-db2
ora.oc4j
1ONLINEONLINEdave-db1
ora.oradb.db
1ONLINEONLINEdave-db2Open
2ONLINEONLINEdave-db1Open
ora.scan1.vip
1ONLINEONLINEdave-db1

注意2：

通过crsctlstatresource–t命令，我们可以看到节点1上的ora.LISTENER.lsnr状态是INTERMEDIATE，和NotAllEndpointsRegistered。

[grid@dave-db2admin]$crs_stat-t
NameTypeTargetStateHost
------------------------------------------------------------
ora.DATA.dgora....up.typeONLINEONLINEdave-db1
ora....ER.lsnrora....er.typeONLINEONLINEdave-db1
ora....N1.lsnrora....er.typeONLINEONLINEdave-db1
ora.asmora.asm.typeONLINEONLINEdave-db1
ora.cvuora.cvu.typeONLINEONLINEdave-db1
ora....SM2.asmapplicationONLINEONLINEdave-db1
ora....B1.lsnrapplicationONLINEONLINEdave-db1
ora....db1.gsdapplicationOFFLINEOFFLINE
ora....db1.onsapplicationONLINEONLINEdave-db1
ora....db1.vipora....t1.typeONLINEONLINEdave-db1
ora....SM1.asmapplicationONLINEONLINEdave-db2
ora....B2.lsnrapplicationONLINEONLINEdave-db2
ora....db2.gsdapplicationOFFLINEOFFLINE
ora....db2.onsapplicationONLINEONLINEdave-db2
ora....db2.vipora....t1.typeONLINEONLINEdave-db2
ora.gsdora.gsd.typeOFFLINEOFFLINE
ora....networkora....rk.typeONLINEONLINEdave-db1
ora.oc4jora.oc4j.typeONLINEONLINEdave-db1
ora.onsora.ons.typeONLINEONLINEdave-db1
ora.oradb.dbora....se.typeONLINEONLINEdave-db2
ora....ry.acfsora....fs.typeONLINEONLINEdave-db1
ora.scan1.vipora....ip.typeONLINEONLINEdave-db1

Listener.ora 文件内容如下：

[grid@dave-db1admin]$morelistener.ora
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))#lineaddedbyAgent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))#lineaddedbyAgent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON#lineaddedbyAgent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON#lineaddedbyAgent

二．可能原因

在MOS的文档中有说明：

Listener in INTERMEDIATE status with"Not All Endpoints Registered" [ID 1454439.1]

使用crsctl stat res -t，监听状态显示为INTERMEDIATE。

ora.LISTENER.lsnr
ONLINEONLINEracdb1
ONLINEINTERMEDIATEracdb2 Not All Endpoints Registered
ora.LISTENER_SCAN1.lsnr
1 ONLINEINTERMEDIATE racdb2 Not All Endpoints Registered

可能的原因如下：

（1）The problem is caused byanother listener defined statically in listener.ora, using the same port and IPis running from the RDBMS ORACLE_HOME, started manually causing the defaultlistener starting from GRID_HOME can not register its endpoint. Hence the errorreported in dbca.

--在ORACLE_HOME的目录下配置了listener.ora，并且使用了相同的IP和端口。在手动的在GRID_HOME下启动默认的监听，就会导致不能注册endpoint.

ps -ef | grep tns:

grid 7222 1 0 Apr26 ? 00:00:13 /u01/app/11.2.0/grid/bin/tnslsnrLISTENER_SCAN1 -inherit
grid 7237 1 0 Apr26 ? 00:00:13 /u01/app/11.2.0/grid/bin/tnslsnrLISTENER -inherit
oracle 7354 1 0 Apr26 ? 00:00:01 /u02/app/oracle/product/11.2.0/db/bin/tnslsnrLISTENER -inherit

我们这里的情况和这里的很类似，不过我们是启动了2个grid下的监听。

(2) Another possible cause is thelistener or scan listener being defined manually in listener.ora, for example:

--另一种可能是listener 或者scan listener 被手工定义在listener.ora中.

LISTENER_SCAN3 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = racnode1)(PORT = 1523))
)

LISTENER_SCAN1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = racnode1)(PORT = 1521))

注意，这种问题只存在Oracle 11gR2 以后的版本。因为11gR2以后，Oracle 的监听配置发生了变化。 这里了解一下即可。关于11gR2 RAC监听的配置，会在单独文章说明。

From 11.2onwards, all listeners should be runing from GRID_HOME, listener andlistener_scan<n> entry should be added automatically into listener.ora,no manual editing is required for TCP definition.

--从Oracle 11gR2 之后，所有的监听都配置在GRID_HOME下，listener 和listener_scan<n> entry 会自动配置到grid 安装用户的listener.ora文件中，而不需要手工配置相关的信息。

三．解决方法

MOS文档提供的解决方法：

1. Stop the listener running from RDBMSORACLE_HOME

$<RDBMS ORACLE_HOME>/bin/lsnrctl stopLISTENER

2. stop the listener from GRID_HOME

$<GRID_HOME>/bin/srvctl stop listener-n <node name>
$<GRID_HOME>/bin/srvctl stop scan_listener -i <scan#>

eg:

$<GRID_HOME>/bin/srvctl stop listener-n racnode1
$<GRID_HOME>/bin/srvctl stop scan_listener -i 1

If above command fails to stop the tnslsnrprocess, please use "kill -9 <pid of tnslsnr>" to stop theLISTENER and LISTENER_SCAN1 process.

3. remove any manually added LISTENER definitionfrom listener.ora if it exists

4. restart the LISTENER andLISTENER_SCAN1 from GRID_HOME

$<GRID_HOME>/bin/srvctl startlistener -n <node name>
$<GRID_HOME>/bin/srvctl start scan_listener -i <scan#>

5. check crsctl stat res -t output, theyboth should show ONLINE status now.

结合前面的说明，我们这里最大的可能是重复存在的监听进程。所以我们这里只需要kill 一组监听即可。 然后监听就正常了。

这里，我们kill 掉如下2个进程：

grid14380102012?00:02:56/oracle/11.2.0.3/grid/bin/tnslsnrLISTENER-inherit
grid15075102012?00:03:07/oracle/11.2.0.3/grid/bin/tnslsnrLISTENER_SCAN1–inherit

kill之后，监听恢复正常。

[grid@dave-db1~]$lsnrctlstatus

LSNRCTLforLinux:Version11.2.0.3.0-Productionon22-FEB-201319:11:06

Copyright(c)1991,2011,Oracle.Allrightsreserved.

Connectingto(DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUSoftheLISTENER
------------------------
AliasLISTENER
VersionTNSLSNRforLinux:Version11.2.0.3.0-Production
StartDate22-FEB-201319:05:06
Uptime0days0hr.6min.0sec
TraceLeveloff
SecurityON:LocalOSAuthentication
SNMPOFF
ListenerParameterFile/oracle/11.2.0.3/grid/network/admin/listener.ora
ListenerLogFile/oracle/app/grid/diag/tnslsnr/dave-db1/listener/alert/log.xml
ListeningEndpointsSummary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.106.13)(PORT=1521)))
ServicesSummary...
Service"+ASM"has1instance(s).
Instance"+ASM2",statusREADY,has1handler(s)forthisservice...
Service"DAVE"has1instance(s).
Instance"dave1",statusREADY,has1handler(s)forthisservice...
Service"oradb"has1instance(s).
Instance"dave1",statusREADY,has1handler(s)forthisservice...
Service"oradbXDB"has1instance(s).
Instance"dave1",statusREADY,has1handler(s)forthisservice...
Thecommandcompletedsuccessfully

[grid@dave-db1~]$crsctlstatresource-t
--------------------------------------------------------------------------------
NAMETARGETSTATESERVERSTATE_DETAILS
--------------------------------------------------------------------------------
LocalResources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
ora.LISTENER.lsnr
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
ora.asm
ONLINEONLINEdave-db1Started
ONLINEONLINEdave-db2Started
ora.gsd
OFFLINEOFFLINEdave-db1
OFFLINEOFFLINEdave-db2
ora.net1.network
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
ora.ons
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
ora.registry.acfs
ONLINEONLINEdave-db1
ONLINEONLINEdave-db2
--------------------------------------------------------------------------------
ClusterResources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1ONLINEONLINEdave-db1
ora.cvu
1ONLINEONLINEdave-db1
ora.dave-db1.vip
1ONLINEONLINEdave-db1
ora.dave-db2.vip
1ONLINEONLINEdave-db2
ora.oc4j
1ONLINEONLINEdave-db1
ora.oradb.db
1ONLINEONLINEdave-db2Open
2ONLINEONLINEdave-db1Open
ora.scan1.vip
1ONLINEONLINEdave-db1

[grid@dave-db1~]$lsnrctlstatusLISTENER_SCAN1

LSNRCTLforLinux:Version11.2.0.3.0-Productionon22-FEB-201319:12:39

Copyright(c)1991,2011,Oracle.Allrightsreserved.

Connectingto(DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUSoftheLISTENER
------------------------
AliasLISTENER_SCAN1
VersionTNSLSNRforLinux:Version11.2.0.3.0-Production
StartDate22-FEB-201317:48:19
Uptime0days1hr.24min.20sec
TraceLeveloff
SecurityON:LocalOSAuthentication
SNMPOFF
ListenerParameterFile/oracle/11.2.0.3/grid/network/admin/listener.ora
ListenerLogFile/oracle/11.2.0.3/grid/log/diag/tnslsnr/dave-db1/listener_scan1/alert/log.xml
ListeningEndpointsSummary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.106.15)(PORT=1521)))
ServicesSummary...
Service"DAVE"has2instance(s).
Instance"dave1",statusREADY,has1handler(s)forthisservice...
Instance"dave2",statusREADY,has1handler(s)forthisservice...
Service"oradb"has2instance(s).
Instance"dave1",statusREADY,has1handler(s)forthisservice...
Instance"dave2",statusREADY,has1handler(s)forthisservice...
Service"oradbXDB"has2instance(s).
Instance"dave1",statusREADY,has1handler(s)forthisservice...
Instance"dave2",statusREADY,has1handler(s)forthisservice...
Thecommandcompletedsuccessfully

---------------------------------------------------------------------------------------

版权所有，文章允许转载，但必须以链接方式注明源地址，否则追究法律责任!

Skype: tianlesoftware

QQ: tianlesoftware@gmail.com

Email: tianlesoftware@gmail.com

Blog: http://blog.csdn.net/tianlesoftware

Weibo: http://weibo.com/tianlesoftware

Twitter: http://twitter.com/tianlesoftware

Facebook: http://www.facebook.com/tianlesoftware

Linkedin: http://cn.linkedin.com/in/tianlesoftware

分享到：
| 设计模式之建造者模式
评论
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

Magicbox
Global site tag (gtag.js) - Google Analytics