Trouble-Maker Engine

Trouble Engine

The initial version of the trouble engine was written in Perl, and was intended to only be run on RHEL3. My, how times have changed. We are now working on the second version of the system. The new version is being written in Perl to take advantage of more rapid development and more centralized libraries. It will, however, require the Archive-Tar and YAML libraries. The format of the trouble-modules has shifted to being zipped archives in YAML. The job of the trouble engine is to select and apply a problem. It does this via the following process:

System identification - The engine determines what kind of system it is running on to limit the the trouble modules from which it may select. This information may be cached for future runs. At this time, valid systems are specified with --version="string":
Trouble selection - The engine chooses a trouble module and applies it to the system. These modules may be selected with the --selection="string" argument:
- random - For the purpose of self-testing and gaining experience, this mode randomly selects a trouble module and applies it to your system. There is no check to prevent this trouble module from being selected again. This is in line with realistic system scenarios.
- [file] - For the purpose of strict testing and training, you may predefine a set of trouble-modules in a YAML file. In this mode, after choosing the first module, and applying it, that module is marked as inactive, and will not be re-used in the session. Alternately, the argument could refer to a tar file, in which case, that file is loaded as the trouble file
If the --selection="string" command-line argument is not specified, the system will default to random selection.
Trouble parsing - After selecting a trouble module, that module is parsed into its components. These components are sub-selected if required by the module, and a final set of trouble components are selected.
Trouble check - Check the requirements for the trouble module before attempting to run it. If it does not pass, exit and explain why.
Trouble application - At this point, the trouble components are executed in order.
1. All files listed are backed up to the backup directory. The scenario backup script is run to backup key information to the backup directory, passing in the OS version as the sole argument. This information, as well as a list of files that could not be backed up, are stored in a file named 'BACKUP' in the backup directory. The backup directory defaults to /tmp/trouble-maker/backup/ but may be overridden with the --backupdir="string" argument.
2. The scenario description is copied into the rescue directory as a file named 'DESCRIPTION'. The backup directory defaults to /tmp/trouble-maker/rescue/ but may be overridden with the --rescuedir="string" argument.
3. The scenario details are copied into the rescue directory as a file named 'DETAILS'. The backup directory defaults to /tmp/trouble-maker/rescue/ but may be overridden with the --rescuedir="string" argument.
4. The scenario check script is copied into the rescue directory as a file named 'CHECK'. The backup directory defaults to /tmp/trouble-maker/rescue/ but may be overridden with the --rescuedir="string" argument. This is not implimented in all scenarios.
5. The scenario script is copied into the rescue directory as a file named 'TROUBLE-SCRIPT' and executed, passing in the OS version as the sole argument. The backup directory defaults to /tmp/trouble-maker/rescue/ but may be overridden with the --rescuedir="string" argument.
6. The scenario description is presented to the user via standard out.
The exercise of rebooting the server is left as a manual step for the user.

Top

Trouble-Maker Modules

Trouble Modules

A trouble module is a fancy name for a tar file that contains a config file and a few scripts. To work with them, either grab unpacked_kitbag from CVS or download the release and issue mkdir /usr/local/trouble-maker/unpacked_kitbag; /usr/local/trouble-maker/bin/unpack_kitbag.pl /usr/local/trouble-maker/kitbag /usr/local/trouble-maker/unpacked_kitbag.

config.yaml

The config.yaml file controls points of commonality between modules. It contains the following pieces of information:

Backup Files - In order to minimize real system damage, these files will get copied into the backup directory.
Backup Script - In order to minimize real system damage, this optional script will run and output to a file named backup.info in the backup directory. This is intended to help restore environmental state.
Description - One or many descriptions of the type that a user would report to the system administrator. Users do not always understand the system, and may report problems inaccurately. However, for the purpose of this project, all descriptions will, in some way, relate to the trouble scenario.
- This is a sub-select item. Therefore, even if multiple descriptions are listed, only one will be displayed to the user.
Details - This is a long-winded explanation of exactly what is broken and how to fix it. This file exists so that the user may have a reference if needed.
Check Script - This script can be run to verify that the problem was actually fixed. Its use is optional but strongly encouraged. It may be written in whatever language you like. At the current time most modules do not impliment this feature.
OS Requirements - List of OS/Versions to which the module applies.
Package Requirements - List of package names that must be installed on the system for the trouble module to be valid.
System Requirements - List of system and Perl commands that must return true for the trouble module to be valid. If a command starts with - or ! it indicates a Perl file test. This allows us to test for specific configurations.
Authors - List of authors that have worked on the module.
Trouble Script - Actual script to be executed to cause the problem. It may be written in whatever language you like.

Example


# This file exists as an example.  It does nothing of any interest.
#
# REMEMBER THAT YAML DOESN'T SUPPORT TABS!!!!
#
# ==========================================================================================================
# In order to minimize real system damage, these files will get copied into the backup directory.
backup files:
  - "/path/to/file1"
  - "/path/to/file2"
  - "/path/to/file3"
# This script will run and output to a file named backup.info in the backup directory.  Its use is optional but strongly encouraged.
backup script: donothing.sh
# One or many descriptions of the type that a user would report to the system administrator.
description:
  - description 1
  - description 2
# This is a long-winded explanation of exactly what is broken and how to fix it. This file only exists if the user decides to give up and needs a reference.
details: If this were a real script, it would have deleted all your files, caused various things to explode, and made prank calls to the local pizzaria.  To fix it, do nothing.
# This script can be run to verify that the problem was actually fixed. Its use is optional but strongly encouraged.
check script: donothing.sh
# List of OS/Versions to which the module applies.
os requirements:
  - RHEL_3
  - Fedora_2
  - SUSE_9
# These packages must be installed for this package to be valid
package requirements:
  COMMON:
    - nopackage
  RHEL_3:
    - httpd
    - bash
    - vsftpd
  Fedora_2:
    - vsftpd
# These system calls must return true in order for this package to be valid, a - or ! preficing the the command indicates a perl file test, may need to enclose in quotes
system requirements:
  COMMON:
    - '! -e "/tmp/bob"'
  RHEL_3:
    - cat /dev/null
  Fedora_2:
    - cat /dev/null
  SUSE_9:
    - '-d "/tmp"'
# Actual trouble script
trouble script: donothing.sh

Tips

YAML requires certain items to be escaped. To check that your YAML file is correct, run tools/check-trouble-config.pl on your YAML file.
To test, use tools/pack_kitbag.pl to pack up all the troubles to your kitbag location.
If your module doesn't do what you expect, edit trouble-maker.pl and %debug variable to turn on debugging.
All modules do not work on all distros. Add what you can, we'll help out with the rest.
Once you have a working module, let us know on the module forum, and we'll start the official testing process for inclusion.

Top