前言
客户11204单机到单机的linux active dataguard环境,之前adg同步都正常,反应从昨晚开始备库突然没有实时同步。备库MRP进程一直处于等待归档日志,而非在线日志。在主库手动切换归档,备库MRP进程才会应用新生成归档日志。
从主库侧看下当前的ADG状态,V$ARCHIVE_DEST_STATUS.RECOVERY_MODE显示是MANAGED REAL TIME APPLY ,直观理解是备库实时应用中,但gap_status='LOG SWITCH GAP‘,明显有问题。
在主库检查可以看到主备延迟挺大的,正常应该少于60s。
THREAD# DEST_ID DEST_NAME TARGET DATABASE_MODE STATUS ERROR RECOVERY_MODE DB_UNIQUE_NAME DESTINATION GAP_STATUS CURRENT_SEQ# LAST_ARCHIVED APPLIED_SEQ# APPLIED_SCN -------- -------- -------------------- -------------------- -------------------- ---------- ---------- -------------------- --------------- --------------- ---------- ---------------- ---------------- ---------------- ---------------- 1 1 LOG_ARCHIVE_DEST_1 LOCAL PRIMARY OPEN VALID IDLE NONE /u03/arch 4317 4316 0 1 2 LOG_ARCHIVE_DEST_2 PHYSICAL STANDBY OPEN_READ-ONLY VALID MANAGED REAL TIME AP ora9 ora9 LOG SWITCH 4317 4316 4316 14788927220612 PLY GAP $ ora dglag status: VALID DG Lag: 6859s查看备库的alert日志:
Thu Oct 22 11:07:18 2020 Managed Standby Recovery starting Real Time Apply Parallel Media Recovery started with 4 slaves Waiting for all non-current ORLs to be archived... All non-current ORLs have been archived. Media Recovery Waiting for thread 1 sequence 4328 Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION Thu Oct 22 11:07:47 2020 RFS[2]: Selected log 7 for thread 1 sequence 4328 dbid 681308804 branch 940761424 Thu Oct 22 11:07:47 2020 Archived Log entry 3731 added for thread 1 sequence 4328 ID 0x2afda2d8 dest 1: Thu Oct 22 11:07:48 2020 Media Recovery Log /u01/arch/1_4328_940761424.dbf Media Recovery Waiting for thread 1 sequence 4329备库MRP进程进程等待归档4329。这里明显有问题,如果实时同步的话,日志应该是类似:
Media Recovery Waiting for thread 1 sequence 4329 (in transit)以是一个ADG主备库的实时同步示意图:
■ 传输流程:
- 主库的LGWR进程写自己的在线重做日志(ORL) - 主库的DataGuard进程LNS(LogWriter Network Process)从SGA的redo buffer里读redo信息然后通过网络服务传输给备用数据库。 - 由LNS传输的redo记录由备用数据库的另外一个DataGuard进程RFS(Remote File Server)接收。 - RFS接收redo记录之后将其写入备用重做日志(SRL)。
■ ARCH or LNS
在备库故障或网络中断期间,DataGuard在主库上使用ARCH进程连续ping备库来确定其状态。当与备库的通信还原后,ARCH ping进程会查询备库控制文件来确定备用数据库最后一个接收到的完整日志文件,以此确定需要从主库传输哪些日志文件来重新同步备库,然后开始使用ARCH进程传输相应文件。
Oracle10g开始默认归档进程数为2,从而在自动处理日志文件间隔时,不影响主数据库的归档操作。
在接下来执行日志切换时,LNS会试图连接备用数据库,成功后也会开始传输当前重做数据,同一时间ARCH进程在后台处理日志文件间隔。
当应用进程(MRP)赶上进度之后,不再读取归档日志文件,转而读取SRL文件。
■ 应用服务(MRP)
MRP进程在库自动应用redo记录来维护与主库的同步,允许对数据的事务性一致访问。默认应用服务需要等待SRL归档之后才应用redo,当然也可以启动实时应用,允许应用服务应用当前SRL的redo记录。
主库的进程如下,没有发现负责传输的LNS进程
13:23:12 sys@ORA9 > SELECT PROCESS, STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS ,CLIENT_PID, pid FROM gV$MANAGED_STANDBY WHERE THREAD#!=0 ORDER BY THREAD#, SEQUENCE#; PROCESS STATUS THREAD# SEQUENCE# BLOCK# BLOCKS CLIENT_PID PID --------- ---------- -------------------- ----------- -------------------- -------------------- ---------------------------------------- -------------------- LGWR CLOSING 1 4183 1352302 16 2663 2663 ARCH CLOSING 1 4320 8192 263 2685 2685 ARCH CLOSING 1 4328 6144 1484 2715 2715 ARCH CLOSING 1 4328 1 7627 2711 2711后台也没有发现LNS进程
$ ps -ef|grep -v grep|grep -E "ora_lns|ora_nsa|ora_nss" $备库进程如下,有MRP进程(wait for log),没有发现RFS进程工作,也可以理解为当前没有数据传输 ing 。
INST_ID PROCESS PID CLIENT_PROCESS CLIENT_PID STATUS THREAD# SEQUENCE# BLOCK# BLOCKS ---------------- --------- ---------------- --------------- --------------- --------------- -------- --------- ---------------- ---------------- 1 ARCH 44113 ARCH 44113 CLOSING 1 4328 6144 1484 1 MRP0 44474 N/A N/A WAIT_FOR_LOG 1 4329 0 0明显备库当前处于10g年代经典的归档同步 ,只有主库发生归档传输时,主库的ARCH进程会把归档传输到备库,备库MRP进程应用归档文件来同步。
2.备库如果是实时同步是什么情况
利用standby redo log启动实时日志模式。备库alert显示: Primary database is in MAXIMUM PERFORMANCE mode. Media Recovery Log /u01/arch/1_485_940761424.dbf Media Recovery Log /u01/arch/1_486_940761424.dbf Media Recovery Log /u01/arch/1_487_940761424.dbf Media Recovery Log /u01/arch/1_488_940761424.dbf Media Recovery Waiting for thread 1 sequence 489 (in transit) 备库MRP一直在实时应用日志 col BLOCKS for 999999999999999 col client_process for a15 col CLIENT_PID for a15 col STATUS for a15 SELECT INST_ID, PROCESS, PID, CLIENT_PROCESS, CLIENT_PID, STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM GV$MANAGED_STANDBY WHERE THREAD#!=0 ORDER BY THREAD#, SEQUENCE#; INST_ID PROCESS PID CLIENT_PROCESS CLIENT_PID STATUS THREAD# SEQUENCE# BLOCK# BLOCKS --------- --------- ---------------- --------------- --------------- --------------- -------- --------- ---------------- ---------------- 1 ARCH 143191 ARCH 143191 CLOSING 1 5599 1 6 1 ARCH 143189 ARCH 143189 CLOSING 1 5600 3473408 1446 1 ARCH 143195 ARCH 143195 CLOSING 1 5602 3682304 553 1 RFS 144485 LGWR 204820 IDLE 1 5603 1020933 3 1 MRP0 144706 N/A N/A APPLYING_LOG 1 5603 1020935 4194304 主库查询备用数据库应用模式 SELECT RECOVERY_MODE FROM V$ARCHIVE_DEST_STATUS WHERE DEST_ID=2; RECOVERY_MODE ----------------------- MANAGED REAL TIME APPLY 如果是非实时,则显示为MANAGEDWe have to make sure that: - the standby redo logs are the same size as the online redo logs, and - there are (( # of online logs per thread + 1) * # of threads) standby redo logs.
检查主库的redo logs和standby redo logs:
ONLINE_GROUP# THREAD# SEQUENCE# BYTES ARC STATUS ---------------- ---------------- ----------- ---------------- --- ---------- 1 1 4325 1073741824 YES INACTIVE 2 1 4326 1073741824 YES INACTIVE 3 1 4327 1073741824 YES INACTIVE 4 1 4328 1073741824 YES INACTIVE 5 1 4329 1073741824 NO CURRENT 6 1 4324 1073741824 YES INACTIVE STANDBY_GROUP# THREAD# SEQUENCE# BYTES USED ARC STATUS ---------------- ---------------- ----------- ---------------- ---------------- --- ---------- 7 1 0 1073741824 0 YES UNASSIGNED 8 1 0 1073741824 0 YES UNASSIGNED 9 1 0 1073741824 0 YES UNASSIGNED 10 0 0 1073741824 0 YES UNASSIGNED 11 0 0 1073741824 0 YES UNASSIGNED 12 0 0 1073741824 0 YES UNASSIGNED 13 0 0 1073741824 0 YES UNASSIGNED 7 rows selected.检查备库的redo logs和standby redo logs:
ONLINE_GROUP# THREAD# SEQUENCE# BYTES ARC STATUS ---------------- ---------------- ----------- ---------------- --- ---------- 1 1 4325 1073741824 YES CLEARING 2 1 4326 1073741824 YES CLEARING 3 1 4327 1073741824 YES CLEARING 4 1 4328 1073741824 YES CLEARING 5 1 4329 1073741824 YES CURRENT 6 1 4324 1073741824 YES CLEARING STANDBY_GROUP# THREAD# SEQUENCE# BYTES USED ARC STATUS ---------------- ---------------- ----------- ---------------- ---------------- --- ---------- 7 1 0 1073741824 0 NO UNASSIGNED 8 1 0 1073741824 0 NO UNASSIGNED 9 1 0 1073741824 0 NO UNASSIGNED 10 1 0 1073741824 0 NO UNASSIGNED 11 1 0 1073741824 0 NO UNASSIGNED 12 1 0 1073741824 0 NO UNASSIGNED 13 1 0 1073741824 0 NO UNASSIGNED 7 rows selected.明显主备库的redo大小和group一致,standby redo log配置正确。备库的standby redo log没发挥作用,空闲中。
现在只知道主库的LSN进程没有启动,备库因而也没有RFS进程。
继续尝试重启备库,重配主库的log_archive_dest_2、log_archive_dest_state_2再让它生效都没效果。
我们回到主库身上,研究alert日志,找找LSN进程哪去了,有没有trc生成,终于有所发现:
主库实例启动时间:2020/09/23 12:24:59 然后我们发现NSA2进程有很多次hung死,再启动,再hung死,再启动的记录。 按每次NSA2进程的pid关键字我们可以得到一个完整的跟踪: Thu Sep 24 00:56:38 2020 NSA2 started with pid=23, OS id=21372 Fri Sep 25 01:02:12 2020 WARN: ARC0: Terminating pid 21372 hung on an I/O operation Fri Sep 25 01:02:41 2020 Killing 1 processes with pids 21372 (Process by index) in order to remove hung processes. Requested by OS process 2685 Fri Sep 25 01:02:41 2020 WARN: ARC2: Terminating pid 21372 hung on an I/O operation ARC2: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Sat Sep 26 01:36:50 2020 NSA2 started with pid=37, OS id=61374 Tue Sep 29 00:21:31 2020 WARN: ARC2: Terminating pid 61374 hung on an I/O operation Tue Sep 29 00:22:01 2020 WARN: ARC0: Terminating pid 61374 hung on an I/O operation Tue Sep 29 00:22:31 2020 Killing 1 processes with pids 61374 (Process by index) in order to remove hung processes. Requested by OS process 2685 Tue Sep 29 00:22:31 2020 Killing 1 processes with pids 61374 (Process by index) in order to remove hung processes. Requested by OS process 2713 ARC2: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Tue Sep 29 00:22:33 2020 NSA2 started with pid=35, OS id=96811 Wed Sep 30 00:21:12 2020 WARN: ARC2: Terminating pid 96811 hung on an I/O operation Wed Sep 30 00:21:42 2020 Killing 1 processes with pids 96811 (Process by index) in order to remove hung processes. Requested by OS process 2713 ARC2: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Wed Sep 30 00:22:07 2020 NSA2 started with pid=17, OS id=115252 Sat Oct 10 00:11:23 2020 WARN: ARC2: Terminating pid 115252 hung on an I/O operation Sat Oct 10 00:11:53 2020 WARN: ARC0: Terminating pid 115252 hung on an I/O operation Sat Oct 10 00:12:23 2020 Killing 1 processes with pids 115252 (Process by index) in order to remove hung processes. Requested by OS process 2685 Sat Oct 10 00:12:23 2020 Killing 1 processes with pids 115252 (Process by index) in order to remove hung processes. Requested by OS process 2713 ARC2: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Sat Oct 10 00:12:24 2020 NSA2 started with pid=17, OS id=10543 Sat Oct 17 00:21:16 2020 WARN: ARC2: Terminating pid 10543 hung on an I/O operation Sat Oct 17 00:21:46 2020 WARN: ARC0: Terminating pid 10543 hung on an I/O operation Sat Oct 17 00:22:16 2020 Killing 1 processes with pids 10543 (Process by index) in order to remove hung processes. Requested by OS process 2685 Sat Oct 17 00:22:16 2020 Killing 1 processes with pids 10543 (Process by index) in order to remove hung processes. Requested by OS process 2713 ARC2: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Sat Oct 17 00:22:18 2020 NSA2 started with pid=17, OS id=126799 Sat Oct 17 01:10:03 2020 WARN: ARC1: Terminating pid 126799 hung on an I/O operation Sat Oct 17 01:10:33 2020 WARN: ARC3: Terminating pid 126799 hung on an I/O operation Sat Oct 17 01:11:03 2020 Killing 1 processes with pids 126799 (Process by index) in order to remove hung processes. Requested by OS process 2715 Sat Oct 17 01:11:03 2020 WARN: ARC0: Terminating pid 126799 hung on an I/O operation ARC0: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Sat Oct 17 01:11:03 2020 ARC1: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Sat Oct 17 01:11:03 2020 NSA2 started with pid=30, OS id=4662 Sat Oct 17 01:36:03 2020 WARN: ARC0: Terminating pid 4662 hung on an I/O operation Sat Oct 17 01:36:33 2020 WARN: ARC1: Terminating pid 4662 hung on an I/O operation Sat Oct 17 01:37:03 2020 Killing 1 processes with pids 4662 (Process by index) in order to remove hung processes. Requested by OS process 2711 Sat Oct 17 01:37:03 2020 WARN: ARC3: Terminating pid 4662 hung on an I/O operation ARC3: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Sat Oct 17 01:37:03 2020 ARC0: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Sat Oct 17 01:37:03 2020 NSA2 started with pid=61, OS id=8520 Thu Oct 22 00:26:28 2020 WARN: ARC0: Terminating pid 8520 hung on an I/O operation Thu Oct 22 00:26:58 2020 WARN: ARC2: Terminating pid 8520 hung on an I/O operation Thu Oct 22 00:27:28 2020 Killing 1 processes with pids 8520 (Process by index) in order to remove hung processes. Requested by OS process 2713 Thu Oct 22 00:27:28 2020 Killing 1 processes with pids 8520 (Process by index) in order to remove hung processes. Requested by OS process 2685 ARC2: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 ARC0: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2 Thu Oct 22 00:29:40 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_mmon_2671.trc (incident=161027): ORA-00445: background process "m000" did not start after 120 seconds Incident details in: /u01/app/oracle/diag/rdbms/stdora9/stdora9/incident/incdir_161027/stdora9_mmon_2671_i161027.trc Thu Oct 22 00:29:42 2020 Dumping diagnostic data in directory=[cdmp_20201022002942], requested by (instance=1, osid=2671 (MMON)), summary=[incident=161027]. Thu Oct 22 00:32:11 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_arc0_2685.trc (incident=161050): ORA-00445: background process "NSA2" did not start after 120 seconds Incident details in: /u01/app/oracle/diag/rdbms/stdora9/stdora9/incident/incdir_161050/stdora9_arc0_2685_i161050.trc Thu Oct 22 00:32:13 2020 Dumping diagnostic data in directory=[cdmp_20201022003213], requested by (instance=1, osid=2685 (ARC0)), summary=[incident=161050]. Thu Oct 22 00:33:34 2020 Thread 1 advanced to log sequence 4311 (LGWR switch) Current log# 5 seq# 4311 mem# 0: /u01/oradata/ora9/redo05.log Thu Oct 22 00:34:28 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_smco_6340.trc (incident=162482): ORA-00445: background process "W000" did not start after 120 seconds Incident details in: /u01/app/oracle/diag/rdbms/stdora9/stdora9/incident/incdir_162482/stdora9_smco_6340_i162482.trc Thu Oct 22 00:34:30 2020 Dumping diagnostic data in directory=[cdmp_20201022003430], requested by (instance=1, osid=6340 (SMCO)), summary=[incident=162482]. Thu Oct 22 00:34:47 2020 Timed out trying to start process m000. Thu Oct 22 00:37:04 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_smco_6340.trc (incident=162483): ORA-00445: background process "W000" did not start after 120 seconds Incident details in: /u01/app/oracle/diag/rdbms/stdora9/stdora9/incident/incdir_162483/stdora9_smco_6340_i162483.trc Thu Oct 22 00:37:06 2020 Dumping diagnostic data in directory=[cdmp_20201022003706], requested by (instance=1, osid=6340 (SMCO)), summary=[incident=162483]. Thu Oct 22 00:39:22 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_mmon_2671.trc (incident=161028): ORA-00445: background process "m000" did not start after 120 seconds Incident details in: /u01/app/oracle/diag/rdbms/stdora9/stdora9/incident/incdir_161028/stdora9_mmon_2671_i161028.trc Thu Oct 22 00:39:23 2020 Dumping diagnostic data in directory=[cdmp_20201022003923], requested by (instance=1, osid=2671 (MMON)), summary=[incident=161028]. Thu Oct 22 00:41:40 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_smco_6340.trc (incident=162484): ORA-00445: background process "W000" did not start after 120 seconds Thu Oct 22 00:44:11 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_mmon_2671.trc (incident=161029): ORA-00445: background process "m000" did not start after 120 seconds Thu Oct 22 00:46:42 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_smco_6340.trc (incident=162485): ORA-00445: background process "W000" did not start after 120 seconds Thu Oct 22 00:48:59 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_mmon_2671.trc (incident=161030): ORA-00445: background process "m000" did not start after 120 seconds Thu Oct 22 00:51:17 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_smco_6340.trc (incident=162486): ORA-00445: background process "W000" did not start after 120 seconds Thu Oct 22 00:53:34 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_mmon_2671.trc (incident=161031): ORA-00445: background process "m000" did not start after 120 seconds Thu Oct 22 00:55:51 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_smco_6340.trc (incident=162487): ORA-00445: background process "W000" did not start after 120 seconds Thu Oct 22 00:58:08 2020 Errors in file /u01/app/oracle/diag/rdbms/stdora9/stdora9/trace/stdora9_mmon_2671.trc (incident=161032): ORA-00445: background process "m000" did not start after 120 seconds Thu Oct 22 00:58:22 2020在22日早上NSA2进程再次由于IO hung死后,没有像以前那样再次启动。这也导致实时redo无法再传输到备库了!
到底IO出了什么问题,我们看下linux的监控:
$ sar Linux 2.6.32-696.10.1.el6.x86_64 (Server-a9e6ca44-974e-49d8-821f-47420a8812ba) 10/22/2020 _x86_64_ (4 CPU) 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 1.95 0.00 3.52 35.04 0.19 59.29 12:20:01 AM all 1.36 0.00 1.10 28.73 0.07 68.74 12:30:01 AM all 0.49 0.00 0.31 77.27 0.03 21.91 12:40:01 AM all 0.98 0.00 0.49 98.51 0.03 0.00 12:50:01 AM all 0.73 0.00 0.30 98.95 0.03 0.00 01:00:01 AM all 2.13 0.00 0.90 85.06 0.05 11.86 01:10:01 AM all 1.86 0.00 1.18 13.87 0.07 83.02 01:20:01 AM all 1.54 0.00 1.51 10.04 0.09 86.83 01:30:02 AM all 1.53 0.00 3.35 26.59 0.17 68.35 01:40:01 AM all 8.17 0.00 0.85 20.07 0.06 70.85 01:50:01 AM all 1.28 0.00 0.44 0.99 0.04 97.25 02:00:01 AM all 1.06 0.00 0.36 0.31 0.03 98.23 02:10:02 AM all 1.94 0.00 0.63 2.20 0.04 95.18每天0点是数据库的备份时间,估计就是由于备份将主库服务器IO占满,导致NSA2进程被ARCH杀掉。
我们尝试杀掉主库的ARCH进程,看ARCH进程重启后会不会带起NSA2进程:
$ ps -ef|grep arc oracle 2711 1 0 Sep23 ? 00:00:55 ora_arc1_stdora9 oracle 2713 1 0 Sep23 ? 00:00:48 ora_arc2_stdora9 oracle 2715 1 0 Sep23 ? 00:01:05 ora_arc3_stdora9 oracle 106728 104206 0 13:59 pts/4 00:00:00 grep arc $ ps -ef|grep arc oracle 2711 1 0 Sep23 ? 00:00:55 ora_arc1_stdora9 oracle 2713 1 0 Sep23 ? 00:00:48 ora_arc2_stdora9 oracle 2715 1 0 Sep23 ? 00:01:05 ora_arc3_stdora9 oracle 106769 104206 0 13:59 pts/4 00:00:00 grep arc $ ps -ef|grep -v grep|grep -E "ora_lns|ora_nsa|ora_nss" oracle 106807 1 0 13:59 ? 00:00:00 ora_nsa2_stdora9对应的alert日志:
Thu Oct 22 13:59:54 2020 ARC2: Detected ARCH process failure ARC2: STARTING ARCH PROCESSES Thu Oct 22 13:59:54 2020 ARC0 started with pid=131, OS id=106797 ARC0: Archival started ARC2: STARTING ARCH PROCESSES COMPLETE ARC0: Becoming the 'no FAL' ARCH ARC0: Becoming the 'no SRL' ARCH Thu Oct 22 13:59:55 2020 NSA2 started with pid=270, OS id=106807 Thu Oct 22 14:00:01 2020 Thread 1 advanced to log sequence 4330 (LGWR switch) Current log# 6 seq# 4330 mem# 0: /u01/oradata/ora9/redo06.log LNS: Standby redo logfile selected for thread 1 sequence 4330 for destination LOG_ARCHIVE_DEST_2 Thu Oct 22 14:00:09 2020 Archived Log entry 7841 added for thread 1 sequence 4329 ID 0x2afda2d8 dest 1: Thu Oct 22 14:00:10 2020 ARC1: Standby redo logfile selected for thread 1 sequence 4329 for destination LOG_ARCHIVE_DEST_2备库的状态:
GROUP# THREAD# SEQUENCE# BLOCKSIZE BYTES USED FIRST_CHANGE# FIRST_TIME LAST_CHANGE# LAST_TIME ARC STATUS -------- -------- --------- ---------------- ---------------- ---------------- ---------------- ------------ ---------------- ------------ --- ---------- 7 1 4330 512 1073741824 10868224 14788927622220 22-OCT-20 14788927625675 22-OCT-20 YES ACTIVE 8 1 0 512 1073741824 0 NO UNASSIGNED 9 1 0 512 1073741824 0 NO UNASSIGNED 10 1 0 512 1073741824 0 NO UNASSIGNED 11 1 0 512 1073741824 0 NO UNASSIGNED 12 1 0 512 1073741824 0 NO UNASSIGNED 13 1 0 512 1073741824 0 NO UNASSIGNED INST_ID PROCESS PID CLIENT_PROCESS CLIENT_PID STATUS THREAD# SEQUENCE# BLOCK# BLOCKS ---------------- --------- ---------------- --------------- --------------- --------------- -------- --------- ---------------- ---------------- 1 ARCH 44113 ARCH 44113 CLOSING 1 4329 1110016 211 1 RFS 44789 LGWR 106807 IDLE 1 4330 21224 4 1 MRP0 44474 N/A N/A APPLYING_LOG 1 4330 21227 2097152
