[BACK]Return to database.sgml CVS log [TXT][DIR] Up to [Development] / failsafe / FailSafe-books / LnxFailSafe_PG

File: [Development] / failsafe / FailSafe-books / LnxFailSafe_PG / database.sgml (download)

Revision 1.1, Wed Nov 29 22:01:12 2000 UTC (16 years, 10 months ago) by vasa
Branch: MAIN
CVS Tags: HEAD

New documentation files for the Programmers' Guide.

<!-- Fragment document type declaration subset:
ArborText, Inc., 1988-1997, v.4001
<!DOCTYPE SET PUBLIC "-//Davenport//DTD DocBook V3.0//EN" [
<!ENTITY scriptlib.sgml SYSTEM "scriptlib.sgml">
<!ENTITY scriptlibapp.sgml SYSTEM "scriptlibapp.sgml">
<!ENTITY startgui.sgml SYSTEM "startgui.sgml">
<!ENTITY preface.sgml SYSTEM "preface.sgml">
<!ENTITY overview.sgml SYSTEM "overview.sgml">
<!ENTITY action.sgml SYSTEM "action.sgml">
<!ENTITY failover.sgml SYSTEM "failover.sgml">
<!ENTITY install.sgml SYSTEM "install.sgml">
<!ENTITY gloss.sgml SYSTEM "gloss.sgml">
<!ENTITY index.sgml SYSTEM "index.sgml">
<!ENTITY monitor SYSTEM "figures/monitor.eps" NDATA eps>
<!ENTITY resource.ai SYSTEM "figures/resource.ai.eps" NDATA eps>
<!ENTITY optional.ai SYSTEM "figures/optional.ai.eps" NDATA eps>
<!ENTITY manager.ai SYSTEM "figures/manager.ai.eps" NDATA eps>
<!ENTITY depend.ai SYSTEM "figures/depend.ai.eps" NDATA eps>
<!ENTITY type.ai SYSTEM "figures/type.ai.eps" NDATA eps>
<!ENTITY attrib.ai SYSTEM "figures/attrib.ai.eps" NDATA eps>
<!ENTITY action.ai SYSTEM "figures/action.ai.eps" NDATA eps>
<!ENTITY star.configuration SYSTEM "figures/star.configuration.eps" NDATA eps>
<!ENTITY n.plus.2.configuration SYSTEM "figures/n.plus.2.configuration.eps" NDATA eps>
<!ENTITY square.configuration SYSTEM "figures/square.configuration.eps" NDATA eps>
]>
-->
<chapter id="LE43007-PARENT">
<title id="LE43007-TITLE">Defining a New Resource Type</title>
<para>This chapter tells you how to define a new resource type:<itemizedlist>
<listitem><para><xref linkend="Z943900385lhj"></para>
</listitem>
<listitem><para><xref linkend="Z943900404lhj"></para>
</listitem>
<listitem><para><xref linkend="Z943900431lhj"></para>
</listitem>
</itemizedlist></para>
<para>It also tells you how to test the results in <xref linkend="Z943901162lhj">.
</para>
<para>To define a new resource type, you must have the following information:<indexterm
id="ITdatabase-0"><primary>resource type</primary><secondary>information required
to define a new resource type</secondary></indexterm></para>
<itemizedlist>
<listitem><para>Name of the resource type. The name can consist of alphanumeric
characters and any of the following:<literallayout>- (hyphen) 
_ (underscore) 
/ 
. 
: 
" 
=
@
, </literallayout></para>
<para>The name cannot contain a space, an unprintable character, or any of
the following characters: <literallayout>*
?
\
#</literallayout></para>
</listitem>
<listitem><para>Name of the cluster to which the resource type will apply.
</para>
</listitem>
<listitem><para>If the resource type is to be restricted to a specific node,
you must know the node name.</para>
</listitem>
<listitem><para>Order of performing the action scripts for resources of this
type in relation to resources of other types:</para>
<itemizedlist>
<listitem><para>Resources are started in the increasing order of this value
</para>
</listitem>
<listitem><para>Resources are stopped in the decreasing order of this value
</para>
<para>Ensure that the number you choose for a new resource type permits the
resource types on which it depends to be started before it is started, or
stopped after it is stopped, as appropriate.</para>
<para><xref linkend="LE10480-PARENT"> shows the conventions used for order
ranges. The values available for customer use are 201-400 and 701-999.</para>
<table frame="topbot" id="LE10480-PARENT">
<title id="LE10480-TITLE">Order Ranges</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="66*">
<colspec colwidth="420*">
<thead>
<row rowsep="1"><entry align="left" valign="bottom"><para>Range</para></entry>
<entry align="left" valign="bottom"><para>Reservation</para></entry></row>
</thead>
<tbody>
<row>
<entry align="left" valign="top"><para>1-100</para></entry>
<entry align="left" valign="top"><para>SGI-provided basic system resource
types, such as <literal>MAC_address</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>101-200</para></entry>
<entry align="left" valign="top"><para>SGI-provided system plug-ins that can
be started before <literal>IP_address</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>201-400</para></entry>
<entry align="left" valign="top"><para>User-defined resource types that can
be started before <literal>IP_address</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>401-500</para></entry>
<entry align="left" valign="top"><para>SGI-provided basic system resource
types, such as <literal>IP_address</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>501-700</para></entry>
<entry align="left" valign="top"><para>SGI-provided system plug-ins that must
be started after <literal>IP_address</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>701-999</para></entry>
<entry align="left" valign="top"><para>User-defined resource types that must
be started after <literal>IP_address</literal></para></entry>
</row>
</tbody>
</tgroup>
</table>
<indexterm id="ITdatabase-1"><primary>resource type</primary><secondary>order
ranges</secondary></indexterm><indexterm id="ITdatabase-2"><primary>order
ranges for resource types</primary></indexterm>
<para><xref linkend="LE30764-PARENT"> shows the order numbers of the resource
types provided with the release or available as plug-ins from SGI.</para>
<table frame="topbot" id="LE30764-PARENT">
<title id="LE30764-TITLE">Resource Type Order Numbers</title>
<tgroup cols="2" colsep="0" rowsep="0">
<colspec colwidth="186*">
<colspec colwidth="300*">
<thead>
<row rowsep="1"><entry align="left" valign="bottom"><para>Order Number</para></entry>
<entry align="left" valign="bottom"><para>Resource Type</para></entry></row>
</thead>
<tbody>
<row>
<entry align="left" valign="top"><para>10</para></entry>
<entry align="left" valign="top"><para><literal>MAC_address</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>20</para></entry>
<entry align="left" valign="top"><para><literal>volume</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>30</para></entry>
<entry align="left" valign="top"><para><literal>filesystem</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>201</para></entry>
<entry align="left" valign="top"><para><literal>NFS</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>401</para></entry>
<entry align="left" valign="top"><para><literal>IP_address</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>411</para></entry>
<entry align="left" valign="top"><para><literal>statd</literal></para></entry>
</row>
<row>
<entry align="left" valign="top"><para>502</para></entry>
<entry align="left" valign="top"><para><literal>Samba</literal></para></entry>
</row>
</tbody>
</tgroup>
</table>
</listitem>
</itemizedlist>
</listitem>
<listitem><para>Restart mode, which can be one of the following values:<indexterm
id="ITdatabase-3"><primary>resource type</primary><secondary>restart mode
</secondary></indexterm> <indexterm id="ITdatabase-4"><primary>restart mode
</primary></indexterm></para>
<itemizedlist>
<listitem><para>0 = Do not restart on monitoring failures</para>
</listitem>
<listitem><para>1 = Restart a fixed number of times</para>
</listitem>
</itemizedlist>
</listitem>
<listitem><para>Number of local restarts (when restart mode is 1).</para>
</listitem>
<listitem><para>Location of the executable script. This is always <?Pub _nolinebreak><filename>
/usr/lib/failsafe/resource_types/<replaceable>resource_type_tname</replaceable></filename><?Pub /_nolinebreak><?Pub Caret> <indexterm
id="ITdatabase-5"><primary><literal>usr/lib/failsafe/resource_types</literal>
directory</primary></indexterm>.</para>
</listitem>
<listitem><para>Monitoring interval, which is the time period (in milliseconds)
between successive executions of the <literal>monitor</literal> action script;
this is only valid for the <literal>monitor</literal> action script.</para>
</listitem>
<listitem><para>Starting time for monitoring. When the resource group is made
online in a cluster node, Linux FailSafe will start monitoring the resources
after the specified time period (in milliseconds).</para>
</listitem>
<listitem><para>Action scripts to be defined for this resource type. You must
specify scripts for <literal>start</literal>, <literal>stop</literal>, <literal>
exclusive</literal>, and <literal>monitor</literal>, although the <literal>
monitor</literal> script may contain only a return-success function if you
wish. If you specify 1 for the restart mode, you must specify a <literal>
restart</literal> script.</para>
</listitem>
<listitem><para>Type-specific attributes to be defined for this resource type.
The action scripts use this information to start, stop, and monitor a resource
of this resource type. For example, NFS requires the following resource keys:
</para>
<itemizedlist>
<listitem><para><literal>export-info</literal> which takes a value that defines
the export options for the file system. These options are used in the <command>
kexportfs</command> command. For example:</para>
<programlisting>export-info = rw,wsync,anon=root</programlisting>
</listitem>
<listitem><para><literal>filesystem</literal> which takes a value that defines
the raw file system. This name is used as input to the <command>mount</command>
command. For example:</para>
<programlisting>filesystem = /dev/xlv/xlv_object</programlisting>
</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
<sect1 id="Z943900385lhj">
<title>Using the GUI</title>
<para>You can use the FailSafe Manager graphical user interface (GUI) to define
a new resource type and to define the dependencies for a given type. For details
about the GUI, see the <citetitle>Linux FailSafe  Administrator's Guide</citetitle>.
For convenience, <xref linkend="Z943897815lhj">, contains information about
starting the GUI.</para>
<sect2>
<title>Define a New Resource Type</title>
<para>To define a new resource type using the GUI, select the following task:<indexterm
id="ITdatabase-6"><primary>resource type</primary><secondary>GUI use</secondary>
</indexterm></para>
<programlisting>Resources &amp; Resource Types => Define a Resource Type</programlisting>
<para>The GUI will prompt you for required and optional information. Online
help is provided for each item.</para>
<para>The following figures show this process for a new resource type called <literal>
newresourcetype</literal>.</para>
<figure id="Z942424847glen">
<title>Select <interface class="menu">Define a New Resource</interface></title>
<graphic entityref="resource.ai"></graphic>
</figure>
<figure id="Z942424883glen">
<title>Specify the Name of the New Resource Type</title>
<graphic entityref="type.ai"></graphic>
</figure>
<figure id="Z942424567glen">
<title>Specify Settings for Required Actions</title>
<graphic entityref="action.ai"></graphic>
</figure>
<figure id="Z942424823glen">
<title>Change Settings for Optional Actions</title>
<graphic entityref="optional.ai"></graphic>
</figure>
<figure id="Z942424617glen">
<title>Set Type-specific Attributes</title>
<graphic entityref="attrib.ai"></graphic>
</figure>
</sect2>
<sect2>
<title>Define Dependencies</title>
<para>To define the dependencies for a given type use the following task:
</para>
<programlisting>Add/Remove Dependencies for a Resource Type</programlisting>
<para><xref linkend="Z942424684glen"> shows an example of adding two dependencies
(<literal>filesystem</literal> and <literal>NFS</literal>) to the <literal>
newresourcetype</literal> resource type.</para>
<figure id="Z942424684glen">
<title id="Z943034219lhj">Add Dependencies</title>
<graphic entityref="depend.ai"></graphic>
</figure>
</sect2>
</sect1>
<sect1 id="Z943900404lhj">
<title>Using <literal>cluster_mgr</literal> Interactively</title>
<para>The following steps show the use of <literal>cluster_mgr</literal> interactively
to define a resource type called <literal>newresourcetype</literal>. Note
that you can have multiple resource types.  For example, if you want to have
some IP addresses that allow local restart (restart mode = 0) and some that
do not (restart mode = 1), you can copy the <literal>IP_address</literal>
type to a new type named <literal>IP_address2</literal> and change just that
value in the <literal>IP_address2</literal>. <indexterm id="ITdatabase-7">
<primary><literal>cmgr</literal>  command</primary></indexterm> <indexterm
id="ITdatabase-8"><primary>resource type</primary><secondary><literal>cluster_mgr
</literal>  use</secondary></indexterm> <indexterm id="ITdatabase-9"><primary><literal>
cluster_mgr</literal>  command</primary></indexterm></para>
<note>
<para>A resource type name cannot contain a space, an unprintable character,
or any of the following characters: <literallayout>* 
? 
\ 
#</literallayout></para>
</note>
<orderedlist>
<listitem><para>Log in as <command>root</command>.</para>
</listitem>
<listitem><para>Execute the <literal>cluster_mgr </literal>command using the <command>
-p</command> option to prompt you for information (the command name can be
abbreviated to <literal>cmgr</literal>): </para>
<programlisting># <userinput>/usr/lib/failsafe/bin/cluster_mgr -p</userinput>&ensp;
Welcome to Linux FailSafe Cluster Manager Command-Line Interface
&nbsp;
cmgr></programlisting>
</listitem>
<listitem><para>Use the <literal>set</literal> subcommand to specify the default
cluster used for <literal>cluster_mgr</literal> operations. In this example,
we use a cluster named <literal>test</literal>:</para>
<programlisting>cmgr> <userinput>set cluster test</userinput>&ensp;</programlisting>
<note>
<para>If you prefer, you can specify the cluster name as needed with each
subcommand.</para>
</note>
</listitem>
<listitem><para>Use the <literal>define resource_type</literal> subcommand.
By default, the resource type will apply across the cluster; if you wish to
limit the resource type to a specific node, enter the node name when prompted.
If you wish to enable restart mode, enter 1 when prompted.</para>
<note>
<para>The following example only shows the prompts and answers for two action
scripts (<literal>start</literal> and <literal>stop</literal>) for a new resource
type named<literal>newresourcetype</literal>.</para>
</note>
<programlisting>cmgr> <userinput>define resource_type newresourcetype</userinput>

(Enter "cancel" at any time to abort)

Node[optional]?
Order ? <userinput>300</userinput>
Restart Mode ? (0)
&nbsp;
DEFINE RESOURCE TYPE OPTIONS
&nbsp;
        1) Add Action Script.
        2) Remove Action Script.
        3) Add Type Specific Attribute.
        4) Remove Type Specific Attribute.
        5) Add Dependency.
        6) Remove Dependency.
        7) Show Current Information.
        8) Cancel. (Aborts command)
        9) Done. (Exits and runs command)
&nbsp;
Enter option:<userinput>1</userinput>
&nbsp;
No current resource type actions
&nbsp;
Action name ? <userinput>start</userinput>
Executable Time? <userinput>40000</userinput>
Monitoring Interval? <userinput>0</userinput>
Start Monitoring Time? <userinput>0</userinput> 
&nbsp;
        1) Add Action Script.
        2) Remove Action Script.
        3) Add Type Specific Attribute.
        4) Remove Type Specific Attribute.
        5) Add Dependency.
        6) Remove Dependency.
        7) Show Current Information.
        8) Cancel. (Aborts command)
        9) Done. (Exits and runs command)
&nbsp;
Enter option:<userinput>1</userinput>
&nbsp;
Current resource type actions:
        Action - 1: start
&nbsp;
Action name <userinput>stop</userinput>
Executable Time? <userinput>40000</userinput>
Monitoring Interval? <userinput>0</userinput>
Start Monitoring Time? <userinput>0</userinput> 
&nbsp;
        1) Add Action Script.
        2) Remove Action Script.
        3) Add Type Specific Attribute.
        4) Remove Type Specific Attribute.
        5) Add Dependency.
        6) Remove Dependency.
        7) Show Current Information.
        8) Cancel. (Aborts command)
        9) Done. (Exits and runs command)
&nbsp;
Enter option:<userinput>3</userinput>
&nbsp;
No current type specific attributes
&nbsp;
Type Specific Attribute ? <userinput>integer-att</userinput>
Datatype ? <userinput>integer</userinput>
Default value[optional] ? <userinput>33</userinput>&ensp;
&nbsp;
        1) Add Action Script.
        2) Remove Action Script.
        3) Add Type Specific Attribute.
        4) Remove Type Specific Attribute.
        5) Add Dependency.
        6) Remove Dependency.
        7) Show Current Information.
        8) Cancel. (Aborts command)
        9) Done. (Exits and runs command)
&nbsp;
Enter option:<userinput>3</userinput>
&nbsp;
Current type specific attributes:
        Type Specific Attribute - 1: export-point
&nbsp;
Type Specific Attribute ? <userinput>string-att</userinput>
Datatype ? <userinput>string</userinput>
Default value[optional] ? <userinput>rw</userinput>
&nbsp;
        1) Add Action Script.
        2) Remove Action Script.
        3) Add Type Specific Attribute.
        4) Remove Type Specific Attribute.
        5) Add Dependency.
        6) Remove Dependency.
        7) Show Current Information.
        8) Cancel. (Aborts command)
        9) Done. (Exits and runs command)
&nbsp;
Enter option:<userinput>5</userinput>
&nbsp;
No current resource type dependencies
&nbsp;
Dependency name ? <userinput>filesystem</userinput>
&nbsp;
        1) Add Action Script.
        2) Remove Action Script.
        3) Add Type Specific Attribute.
        4) Remove Type Specific Attribute.
        5) Add Dependency.
        6) Remove Dependency.
        7) Show Current Information.
        8) Cancel. (Aborts command)
        9) Done. (Exits and runs command)
&nbsp;
Enter option:<userinput>7</userinput>
&nbsp;
Current resource type actions:
        Action - 1: start
        Action - 2: stop
&nbsp;
Current type specific attributes:
        Type Specific Attribute - 1: integer-att
        Type Specific Attribute - 2: string-att
&nbsp;
No current resource type dependencies
&nbsp;
Resource dependencies to be added:
        Resource dependency - 1: filesystem
&nbsp;
        1) Add Action Script.
        2) Remove Action Script.
        3) Add Type Specific Attribute.
        4) Remove Type Specific Attribute.
        5) Add Dependency.
        6) Remove Dependency.
        7) Show Current Information.
        8) Cancel. (Aborts command)
        9) Done. (Exits and runs command)
&nbsp;
Enter option:<userinput>9</userinput>
Successfully defined resource_type newresourcetype
&nbsp;
cmgr> <userinput>show resource_types in cluster test</userinput>
&nbsp;
NFS
template
newresourcetype
statd
MAC_address
IP_address
filesystem
volume
&nbsp;
cmgr> <userinput>exit</userinput>
# </programlisting>
</listitem>
</orderedlist>
</sect1>
<sect1 id="Z943900431lhj">
<title>Using <literal>cluster_mgr</literal> With a Script</title>
<para>You can write a script that contains all of the information required
to define a resource type and supply it to <literal>cluster_mgr</literal>
by using the <command>-f</command> option:<indexterm id="ITdatabase-10"><primary>
resource type</primary><secondary>script use</secondary></indexterm></para>
<literallayout>cluster_mgr -f <replaceable>scriptname</replaceable></literallayout>
<para>Or, you could include the following as the first line of the script
and then execute the script itself:</para>
<programlisting>#!/usr/lib/failsafe/bin/cluster_mgr -f</programlisting>
<para>If any line of the script fails, <literal>cluster_mgr</literal> will
exit. You can choose to ignore the failure and continue the process by using
the -i option, as follows:</para>
<programlisting>#!/usr/lib/failsafe/bin/cluster_mgr -if</programlisting>
<note>
<para>If you include <command>-i</command> when using a cluster_mgr command
line as the first line of the script, you must use this exact syntax (that
is, <command>-if</command>).</para>
</note>
<para><indexterm id="ITdatabase-11"><primary><literal>/usr/lib/failsafe/cmgr-templates/cmgr-create-resource_type
</literal>  directory</primary></indexterm>A template script for creating
a new resource type is located in  <filename>/usr/lib/failsafe/cmgr-templates/cmgr-create-resource_type
</filename> .<indexterm id="ITdatabase-12"><primary>resource type</primary>
<secondary>script templates</secondary></indexterm> <indexterm id="ITdatabase-13">
<primary>templates</primary><secondary>resource type script definition</secondary>
</indexterm> Each line of the script must be a valid <literal>cluster_mgr
</literal> line, a comment line (starting with #), or a blank line.</para>
<note>
<para>You must include a <literal>done</literal> command line to finish a
multi-level command. If you concatenate information from multiple template
scripts to prepare your cluster configuration, you must remove the <literal>
quit</literal> at the end of each template script.</para>
</note>
<para>For example, you could use the following script to define the same <literal>
newresourcetype</literal> resource type defined interactively in the previous
section:</para>
<programlisting># newresourcetype.script: Script to define the "newresourcetype" resource type

set cluster test
define resource_type newresourcetype
set order to 300
set restart_mode to 0
add action start
set exec_time to 40000
set monitor_interval to 0
set monitor_time to 0
done
add action stop
set exec_time to 40000
set monitor_interval to 0
set monitor_time to 0
done
add type_attribute integer-att
set data_type to integer
set default_value to 33
done
add type_attribute string-att
set data_type to string
set default_value to rw
done
add dependency filesystem
done
quit&nbsp;</programlisting>
<para>When you execute the <literal>cluster_mgr -f</literal> command line
with this script, you will see the following output:</para>
<programlisting># <userinput>/usr/lib/failsafe/bin/cluster_mgr -f newresourcetype.script
</userinput>
Successfully defined resource_type newresourcetype

# </programlisting>
<para>To verify that the resource type was defined, enter the following:</para>
<programlisting># /<userinput>usr/lib/failsafe/bin/cluster_mgr -c "show resource_types in cluster test"
</userinput>
&nbsp;
NFS
template
newresourcetype
statd
MAC_address
IP_address
filesystem
volume</programlisting>
</sect1>
<sect1 id="Z943901162lhj">
<title>Testing a New Resource Type</title>
<para>After adding a new resource type, you should test it as follows:</para>
<orderedlist>
<listitem><para>Define a resource group that contains resources of the new
type. Ensure that the group contains all of the resources on which the new
resource type depends.</para>
</listitem>
<listitem><para>Bring the resource group online in the cluster using <command>
cluster_mgr</command> or the GUI.</para>
<para>For example, using <command>cluster_mgr</command>:<programlisting>cmgr> <userinput>
admin online resource_group new_rg in cluster test_cluster</userinput></programlisting></para>
</listitem>
<listitem><para>Check the status of the resource group using <literal>cluster_mgr
</literal> or GUI after a few minutes.</para>
<para>For example:<programlisting>cmgr> <userinput>show status of resource_group new_rg in cluster test_cluster
</userinput>

</programlisting></para>
</listitem>
<listitem><para>If the resource group has been made online successfully, you
will see output similar to the following: <programlisting>State: Online
Error: No error
Owner: node1</programlisting></para>
</listitem>
<listitem><para>If there are resource group errors, do the following:<itemizedlist>
<listitem><para>Check the <literal>srmd</literal> logs ( <filename>/var/log/failsafe/srmd_<replaceable>
nodename</replaceable></filename>) on the node on which the resource group
is online</para>
</listitem>
<listitem><para>Search for the string <literal>ERROR</literal> in the log
file. There should be an error message about a resource in the resource group.
The message also provides information about the action script that failed.
For example: <programlisting>Wed Nov 3 04:20:10.135 &lt;E ha_srmd srm 12127:1 sa_process_tasks.c:627>
CI_FAILURE, ERROR: Action (exclusive) for resource (10.0.2.45) of type
(IP_address) failed with status (failed)
exclusive script failed for the resource 10.0.2.45 of resource type
IP_address. The status "failed"
indicates that the script returned an error.</programlisting></para>
</listitem>
<listitem><para>Check the script logs (<filename>/var/log/failsafe/script_<replaceable>
nodename</replaceable></filename> on the same node) for <literal>IP_address
</literal> <literal>exclusive</literal> script errors.</para>
</listitem>
<listitem><para>After the fixing the problems in the action script, perform
an <literal>offline_force</literal> operation to clear the error. For example: <literallayout>
cmgr> <userinput>admin offline_force resource_group new_rg in cluster test_cluster
</userinput></literallayout></para>
</listitem>
</itemizedlist></para>
</listitem>
</orderedlist>
</sect1>
</chapter>
<?Pub *0000025051>