系统环境：
操作系统： AIX 5300-09
集群软件： CRS 10.2.0.1
数据库： Oracle 10.2.0.1
本案例是用于基于VG Concurrent 的共享存储，通过HACMP 实现卷组的并发
案例分析：
一、错误现象：
1、Oracle 用户无法访问设备文件
2、CRS server启动失败
[oracle@aix211 ~]$ls -l /dev
|
1
2
3
4
5
6
|
/dev/__vg10: No permission
/dev/audit: No permission
/dev/cd0: No permission
/dev/clone: No permission
/dev/console: No permission
/dev/error: No permission
|
查看设备文件属性，发现被改为oracle：dba
[oracle@aix211 ~]$ls -ld /dev
drw-rw---- 6 oracle dba 3584 Sep 16 11:38 /dev
重新更改设备文件属性
|
1
2
3
4
|
[root@aix211 /]#chown root.
system
/dev
[root@aix211 /]#ls -ld /dev
drw-rw---- 6 root
system
3584 Sep 16 11:38 /dev
[root@aix211 /]#chmod 775 /dev
|
Oracle用户可以正常访问设备文件
|
1
2
3
4
5
6
7
8
9
|
[root@aix211 /]#su - oracle
[oracle@aix211 ~]$ls -l /dev
total 24
crw-rw---- 1 root
system
10, 0 Aug 29 2013 IPL_rootvg
srwxrwxrwx 1 root
system
0 Sep 16 10:22 SRC
brw-rw---- 1 oracle dba 88, 9 Sep 11 12:15 control1_1
brw-rw---- 1 oracle dba 88, 10 Sep 11 12:15 control2_2
brw-rw---- 1 oracle dba 88, 11 Sep 11 12:16 control3_3
crw-rw---- 1 root
system
88, 0 Sep 11 12:08 datavg
|
但是CRS server仍然不能正常启动！
二、重新配置CRS:
1、清理ocr和vote disk磁盘信息（两个节点）
|
1
2
3
4
5
6
7
8
9
10
|
[root@aix211 /]#dd
if
=/dev/zero of=/dev/rrac_ocr bs=8192 count=2560
2560+0 records in
2560+0 records out
[root@aix211 /]#dd
if
=/dev/zero of=/dev/rrac_vote bs=8192 count=2560
2560+0 records in
2560+0 records out
[root@aix211 /]#ls -l /dev|grep ocr
brw-rw---- 1 oracle dba 88, 1 Sep 11 12:15 rac_ocr
crw-r----- 1 root oinstall 88, 1 Sep 16 11:05 rrac_ocr
[root@aix211 /]#chown oracle:dba /dev/rrac_ocr
|
2、重新运行root.sh脚本，配置CRS（两个节点）
node1：
[root@aix211 install]#./rootdelete.sh
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
Shutting down Oracle Cluster Ready Services (CRS):
Sep 16 11:48:57.011 | ERR | failed to connect to daemon,
errno
(2)
Stopping resources.
Error
while
stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.
Shutdown has begun. The daemons should
exit
soon.
Checking to see
if
Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script
for
Oracle Cluster Ready services
Updating ocr file
for
downgrade
Cleaning up SCR settings in
'/etc/oracle/scls_scr'
|
[root@aix211 install]#/u01/crs_1/root.sh
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
WARNING: directory
'/u01'
is not owned by root
Checking to see
if
Oracle CRS stack is already configured
Checking to see
if
any 9i GSD is up
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory
'/u01'
is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <
private
interconnect name> <hostname>
node 1: aix211 aix211-priv aix211
node 2: aix212 aix212-priv aix212
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry
for
cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
aix211
CSS is inactive on these nodes.
aix212
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.
|
node2：
[root@aix212 install]#./rootdelete.sh
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
Shutting down Oracle Cluster Ready Services (CRS):
Sep 16 11:48:57.011 | ERR | failed to connect to daemon,
errno
(2)
Stopping resources.
Error
while
stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.
Shutdown has begun. The daemons should
exit
soon.
Checking to see
if
Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script
for
Oracle Cluster Ready services
Updating ocr file
for
downgrade
Cleaning up SCR settings in
'/etc/oracle/scls_scr'
|
[root@aix212@ /]#/u01/crs_1/root.sh
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
|
WARNING: directory
'/u01'
is not owned by root
Checking to see
if
Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory
'/u01'
is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <
private
interconnect name> <hostname>
node 1: aix211 aix211-priv aix211
node 2: aix212 aix212-priv aix212
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry
for
cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
aix211
aix212
CSS is active on all nodes.
Waiting
for
the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent)
for
configuring nodeapps
The given interface(s),
"en0"
is not
public
. Public interfaces should be used to configure
virtual
IPs.
|
在node2上运行vipca，配置vip
@至此，CRS重新配置成功！
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[root@aix212@ /]#crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@aix212@ /]#crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.aix211.gsd application ONLINE ONLINE aix211
ora.aix211.ons application ONLINE ONLINE aix211
ora.aix211.vip application ONLINE ONLINE aix211
ora.aix212.gsd application ONLINE ONLINE aix212
ora.aix212.ons application ONLINE ONLINE aix212
ora.aix212.vip application ONLINE ONLINE aix212
|
三、重新注册Listener和Database
1、注册listener
通过netca工具，重新reconfigure就可以完成listener的注册！
|
1
2
3
4
5
6
7
8
9
10
11
|
[root
@aix212
@ /]#crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....
11
.lsnr application ONLINE ONLINE aix211
ora.aix211.gsd application ONLINE ONLINE aix211
ora.aix211.ons application ONLINE ONLINE aix211
ora.aix211.vip application ONLINE ONLINE aix211
ora....
12
.lsnr application ONLINE ONLINE aix212
ora.aix212.gsd application ONLINE ONLINE aix212
ora.aix212.ons application ONLINE ONLINE aix212
ora.aix212.vip application ONLINE ONLINE aix212
|
2、注册Database和Instance
注册Database：
[root@aix212@ /]#srvctl add database -h
|
1
2
3
4
5
6
7
8
9
10
11
|
Usage: srvctl add database -d <name> -o <oracle_home> [-m <domain_name>] [-p <spfile>] [-A <name|ip>/netmask] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}] [-s <start_options>] [-n <db_name>] [-y {AUTOMATIC | MANUAL}]
-d <name> Unique name
for
the database
-o <oracle_home> ORACLE_HOME
for
cluster database
-m <domain> Domain
for
cluster database
-p <spfile> Server parameter file
for
cluster database
-A <addr_str> Database cluster alias
-n <db_name> Database name (DB_NAME),
if
different
from
the unique name given by the -d option
-r <role> Role of the database (primary, physical_standby, logical_standby)
-s <start_options> Startup options
for
the database
-y <dbpolicy> Management policy
for
the database (automatic, manual)
-h Print usage
|
[root@aix212@ /]#su - oracle
[oracle@aix212@ ~]$srvctl add database -d prod -o $ORACLE_HOME
|
1
2
3
4
5
6
7
8
9
10
11
12
|
[oracle
@aix212
@ ~]$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....
11
.lsnr application ONLINE ONLINE aix211
ora.aix211.gsd application ONLINE ONLINE aix211
ora.aix211.ons application ONLINE ONLINE aix211
ora.aix211.vip application ONLINE ONLINE aix211
ora....
12
.lsnr application ONLINE ONLINE aix212
ora.aix212.gsd application ONLINE ONLINE aix212
ora.aix212.ons application ONLINE ONLINE aix212
ora.aix212.vip application ONLINE ONLINE aix212
ora.prod.db application OFFLINE OFFLINE
|
注册Instance：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
[oracle
@aix212
@ ~]$srvctl add instance -h
Usage: srvctl add instance -d <name> -i <inst_name> -n <node_name>
-d <name> Unique name
for
the database
-i <inst> Instance name
-n <node> Node name
-h Print usage
[oracle
@aix212
@ ~]$srvctl add instance -d prod -i prod1 -n aix211
[oracle
@aix212
@ ~]$srvctl add instance -d prod -i prod2 -n aix212
[oracle
@aix212
@ ~]$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....
11
.lsnr application ONLINE ONLINE aix211
ora.aix211.gsd application ONLINE ONLINE aix211
ora.aix211.ons application ONLINE ONLINE aix211
ora.aix211.vip application ONLINE ONLINE aix211
ora....
12
.lsnr application ONLINE ONLINE aix212
ora.aix212.gsd application ONLINE ONLINE aix212
ora.aix212.ons application ONLINE ONLINE aix212
ora.aix212.vip application ONLINE ONLINE aix212
ora.prod.db application OFFLINE OFFLINE
ora....d1.inst application OFFLINE OFFLINE
ora....d2.inst application OFFLINE OFFLINE
|
通过crs工具启动Database：
[oracle@aix212@ ~]$srvctl start database -d prod
PRKP-1001 : Error starting instance prod1 on node aix211
CRS-0184: Cannot communicate with the CRS daemon.
PRKP-1001 : Error starting instance prod2 on node aix212
CRS-0184: Cannot communicate with the CRS daemon.
启动Instance失败，通过sqlplus手工启动：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[oracle@aix212@ ~]$sqlplus
'/as sysdba'
SQL*Plus: Release
10.2.
0.1.
0
- Production
on
Tue Sep
16
12
:
08
:
10
2014
Copyright (c)
1982
,
2005
, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area
1258291200
bytes
Fixed Size
2020552
bytes
Variable Size
352324408
bytes
Database Buffers
889192448
bytes
Redo Buffers
14753792
bytes
Database mounted.
Database opened.
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[oracle@aix211 aix211]$sqlplus
'/as sysdba'
SQL*Plus: Release
10.2.
0.1.
0
- Production
on
Tue Sep
16
12
:
09
:
37
2014
Copyright (c)
1982
,
2005
, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area
1258291200
bytes
Fixed Size
2020552
bytes
Variable Size
335547192
bytes
Database Buffers
905969664
bytes
Redo Buffers
14753792
bytes
Database mounted.
Database opened.
|
查看crs启动resource信息：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[oracle
@aix211
aix211]$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....
11
.lsnr application ONLINE ONLINE aix211
ora.aix211.gsd application ONLINE ONLINE aix211
ora.aix211.ons application ONLINE ONLINE aix211
ora.aix211.vip application ONLINE ONLINE aix211
ora....
12
.lsnr application ONLINE ONLINE aix212
ora.aix212.gsd application ONLINE ONLINE aix212
ora.aix212.ons application ONLINE ONLINE aix212
ora.aix212.vip application ONLINE ONLINE aix212
ora.prod.db application ONLINE ONLINE aix211
ora....d1.inst application ONLINE ONLINE aix211
ora....d2.inst application ONLINE ONLINE aix212
|
再通过crs工具重新启动Instance：
[oracle@aix211 aix211]$srvctl stop instance -d prod -i prod1
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[oracle@aix211 aix211]$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora...
.11.
lsnr application ONLINE ONLINE aix211
ora.aix211.gsd application ONLINE ONLINE aix211
ora.aix211.ons application ONLINE ONLINE aix211
ora.aix211.vip application ONLINE ONLINE aix211
ora...
.12.
lsnr application ONLINE ONLINE aix212
ora.aix212.gsd application ONLINE ONLINE aix212
ora.aix212.ons application ONLINE ONLINE aix212
ora.aix212.vip application ONLINE ONLINE aix212
ora.prod.db application ONLINE ONLINE aix211
ora....d1.inst application OFFLINE OFFLINE
ora....d2.inst application ONLINE ONLINE aix212
|
[oracle@aix211 aix211]$srvctl start instance -d prod -i prod1
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[oracle@aix211 aix211]$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora...
.11.
lsnr application ONLINE ONLINE aix211
ora.aix211.gsd application ONLINE ONLINE aix211
ora.aix211.ons application ONLINE ONLINE aix211
ora.aix211.vip application ONLINE ONLINE aix211
ora...
.12.
lsnr application ONLINE ONLINE aix212
ora.aix212.gsd application ONLINE ONLINE aix212
ora.aix212.ons application ONLINE ONLINE aix212
ora.aix212.vip application ONLINE ONLINE aix212
ora.prod.db application ONLINE ONLINE aix211
ora....d1.inst application ONLINE ONLINE aix211
ora....d2.inst application ONLINE ONLINE aix212
|
@至此，通过crs工具可以正常启动和关闭Database，由于误操作而引起的血案，抢救成功！
本文转自 客居天涯 51CTO博客，原文链接：http://blog.51cto.com/tiany/1553383，如需转载请自行联系原作者
