OpenSCAP Remediation

The XCCDF standard provides a vendor neutral way to compose a checklist for assessing computer systems. While the XCCDF specification focuses mainly on the scan, there is also a small module which defines how to put the scanned system into the desired (compliant) state. This blogpost introduces a new feature of OpenSCAP, the remediation, which helps you execute remediation command in the automated way.

Your security policy may contain the remediation commands for each Rule within the XCCDF document. Remediation commands are encapsulated within <xccdf:fix> elements. Attributes of the fix element define metadata consumed by scanner tool in the time of execution. For more detailed info about XCCDF Remediation please consult NIST-IR-7275, and search for fix and remediation keywords.

The following snippet is an example of a XCCDF Rule with both remediation and scanning (OVAL) definitions. The system attribute defines the interpreter of the commands, you can use bash, perl, python, or whatever your scanner supports. Obviously, the fix content tends to be unportable across different operating systems, thus the XCCDF allows you to specify multiple fix elements -- each for different OS. These fix elements can be distinguished by diffrerent CPE identifier within the platform attribute. The scanner select the one most suitable for the target system. Other attributes like, disruption level or flag indicating whether the reboot is required, can be used by scanner tool to choose the fix element.

<Rule id="xccdf_moc.elpmaxe.www_rule_1">
    <title> The telnet-server Package Shall Not Be Installed </title>
        Removing the telnet-server package decreases the risk
        of the telnet service's accidental (or intentional) activation.
    <fix platform="cpe:/o:redhat:enterprise_linux:6" reboot="false"
        disruption="low" system="urn:xccdf:fix:script:sh">
        if rpm -q telnet-server; then
            yum -y remove telnet-server
    <check system="">
        <check-content-ref href="test_remediation_simple.oval.xml" name="oval:moc.elpmaxe.www:def:1"/>

Remediation Support in Earlier OpenSCAP

Previously, in OpenSCAP we didn't execute fix elements. We have only allowed export of bash fix scripts to a file. In order to bring your machine to the compliant state, You have need to take the following four steps:
(1) Execute OpenSCAP scan and get xccdf:TestResult stored to some file like:
$ oscap xccdf eval --results ~/my-results-xccdf.xml /usr/share/scap/my-policy-xccdf.xml

(2) Generate bash script from the result file:
$ oscap xccdf generate fix --output ~/ ~/my-results-xccdf.xml

(3) Review the script by eye and execute it.
$ ~/

(4) Execute OpenSCAP scan again and verify applied fixes:
$ oscap xccdf eval --results ~/my-results-xccdf2.xml /usr/share/scap/my-policy-xccdf.xml

Shortcomings of That Approach
For the second step, OpenSCAP library used XSLT transformation to generate scripts. However, some of the features of XCCDF (Text Substitution, CPE applicability, and correct XCCDF Processing) were missing and these features seem to be bordeline impossible to implement with pure XSLT language. Further, when the administrator executed the remediation script manually in the step (3), the output of the script was not saved back to the XCCDF document and the second scan may not be issued. Thus the chain of evidences was discontinued.

The newer version of OpenSCAP library addresses these problems by integrating these 4 steps together. It introduces two new processess which I named Online and Offline remediation.

oscap's Online Remediation
The online remediation executes fix elements in the time of scanning. First of all OpenSCAP evaluates XCCDF as usual, assessing the results by evaluation of OVAL definitions. Next, each Rule which has failed is candidate for remediation. OpenSCAP searches for appropriate fix element, resolves it, prepares the environment and executes the script. Any output of the fix script is captured by OpenSCAP and stored within the result element. The return value of the fix script is stored as well.

Whenever OpenSCAP executes fix script, it immediatelly evaluates the OVAL definition again (to verify that fix script has been applied correctly). In this second run, if the OVAL returns success the result of the Rule is fixed otherwise it is error. The online remediation is executed by --remediate command-line switch.

$ oscap xccdf eval --remediate --results ~/my-results-xccdf.xml /usr/share/scap/my-policy-xccdf.xml

oscap's Offline Remediation
The offline remediation on the other hand allows you to postpone fix execution. In first step, you only evaluate system and store the xccdf:TestResult on your disk:

$ oscap xccdf eval --result ~/my-results-xccdf.xml /usr/share/scap/my-policy-xccdf.xml

In the second step, oscap executes the fix scripts and verifies the result. It is safe to store results into the input file, no data will be lost. During the offline remediation OpenSCAP creates new xccdf:TestResult element which is based on the previous one and inherits all the data. The newly created TestResult differs only in the rule-result elements which have had failed in the first place. For those the remediation is executed.

$ oscap xccdf remediate --results ~/my-results-xccdf.xml ~/my-results-xccdf.xml

Examplary result XCCDF
The following are snippets of the
~/my-results-xccdf.xml. These demostrate what data are exported by OpenSCAP to the result document. In the first example the rule-result failed to remediate, because I had not enough privileges to modify package set on the system.

    <rule-result idref="xccdf_moc.elpmaxe.www_rule_1" time="2013-03-22T19:15:11" weight="1.000000">
      <message severity="info">Fix execution comleted and returned: 1</message>
      <message severity="info">Loaded plugins: auto-update-debuginfo, langpacks, presto, refresh-packagekit
You need to be root to perform this command.
      <message severity="info">Failed to verify applied fix: Checking engine returns: fail</message>
      <fix xmlns:xhtml="" platform="cpe:/o:redhat:enterprise_linux:6"
        reboot="false" disruption="low" system="urn:xccdf:fix:script:sh">
        if rpm -q telnet-server; then
            yum remove -y telnet-server
      <check system="">
        <check-content-ref name="oval:moc.elpmaxe.www:def:1" href="test_remediation_simple.oval.xml"/>

In the second try, I logged in as root user and the remediation succeeded. You can see that OpenSCAP exported full output of the fix script. The output of yum command within the XML file looks clumsy. On the other hand, I believe that in the critical environments the output of remediation command should be properly archieved, especially if the remediation is fully automated.

<rule-result idref="xccdf_moc.elpmaxe.www_rule_1" time="2013-03-22T19:16:03" weight="1.000000">
      <message severity="info">Fix execution comleted and returned: 0</message>
      <message severity="info">Loaded plugins: auto-update-debuginfo, langpacks, presto, refresh-packagekit
Resolving Dependencies
--&gt; Running transaction check
---&gt; Package telnet-server.x86_64 1:0.17-51.fc16 will be erased
--&gt; Finished Dependency Resolution

Dependencies Resolved

Package              Arch          Version                Repository      Size
telnet-server        x86_64        1:0.17-51.fc16        @fedora        53 k

Transaction Summary
Remove  1 Package

Installed size: 53 k
Downloading Packages:
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing    : 1:telnet-server-0.17-51.fc16.x86_64                          1/1
  Verifying  : 1:telnet-server-0.17-51.fc16.x86_64                          1/1

  telnet-server.x86_64 1:0.17-51.fc16

      <fix xmlns:xhtml="" platform="cpe:/o:redhat:enterprise_linux:6"
        reboot="false" disruption="low" system="urn:xccdf:fix:script:sh">
        if rpm -q telnet-server; then
            yum remove -y telnet-server
      <check system="">
        <check-content-ref name="oval:moc.elpmaxe.www:def:1" href="test_remediation_simple.oval.xml"/>

OpenSCAP info module

OpenSCAP scanner 0.9.2 introduces a new function.

In the SCAP world, there is a handful of standards and file formats. The oscap tool can process increasing number of them, but user still needs to know what is what and which option to use. Situation partially improves with emerge of DataStreams, but still DataStreams are rather rare.

If you are unsure how to use a particular file, you can either open and read the file, or you can you use this new info module of OpenSCAP which parses the file and extracts relevant information for you. The info module determines SCAP content type, specification version, date of creation, date of import and so on. Further, for an XCCDF or a Datastream file, info module prints out IDs of incorporated profiles, components, and datastreams. These IDs can be used to specify the target for evaluation. Use options --profile, --xccdf-id (or --oval-id), and --datastream-id respectivelly.

Here is an example of the info module output, when applied to a testing Source DataStream.

    $ oscap info example-sds.xml
    Document type: Source Data Stream
    Imported: 2012-12-01T18:22:50

    Generated: (null)
    Version: 1.2
                Profile: xccdf_moc.elpmaxe.www_profile_1
                Profile: xccdf_moc.elpmaxe.www_profile_2
    No dictionaries.

Example of evaluation using the given info.

    $ oscap xccdf eval \
        --results-arf my-scan-result-arf.xml \
        --datastream-id \
        --xccdf-id \
        --profile xccdf_moc.elpmaxe.www_profile_2 \

OpenSCAP community strives to improve the usability of their scanner. If you have an idea on how to improve the tool or library, don't hesitate and contact them on the mailing list. I did with this info module and it worked. :)

An XCCDF trap into which authors often fell

I have seen enough SCAP contents and products (even the official ones) which contain small deviations from the standard. Today I would like to describe one bug which is the most common as well as it is subtle and hard to spot.

In one of my previous posts I have shown a very simple XCCDF file, it contained only a single rule. If you extend it by adding similar rules you will end up with a single checklist. What is often needed by organizations are multiple checklist, indeed a different policy you will enforce on personal laptop compared to mission critical server. The XCCDF standard addresses this demand and enables you to carry multiple checklists together in a single file. This is achieved by profiles. Each profile may select a different set of rules in document and thus establish a different policy.

This brings us to the first problem, selecting is complex mechanism and selection of a given rule is determined by multiple factors. These factors needs to be applied in correct order as the standard defines. (By the way that's also the reason why I have recently rewritten selector processing in OpenSCAP). Whether an item becomes selected or not is determined by the following factors:

  • default setting of the item (the @selected attribute)
  • settings in the profile or inherited profiles
  • selection of parent group
  • selection of clusters
  • requires/conflicts dependencies.

To make it even more difficult, the XCCDF standard specifies:

An unselected group SHALL NOT be processed, and its contents SHALL NOT be processed either (i.e., all descendants of an unselected group are implicitly unselected).

Which meens that some of your rules might become unselected when the superior group is unselected. This behaviour seems to be inconsistent with what an average content author expects. And even experts might get confused by this rule. Related problems can be even found in USGCB and STIG guidances.

Hence, my humble advice for content authors is: Please consult your checklist with OpenSCAP scanner and make sure that your final policies really enforce what you expect them to enforce.

Spacewalk & OpenSCAP :: Two productizations in two days

On Wednesday, SuSE released a new version of their system manager based on Spacewalk 1.7+ codebase. Being Spacewalk/OpenSCAP integration architect, I was truly delighted to see that their marketing has made a major news from the OpenSCAP integration. What matters the fact, that article has contained a few minor misconceptions from a technical perspective. (I have also learned that it is not a duty of yours competitors marketing to admit which company has lead the project and contributed all of the advertised changes upstream. :-))

On Thursday, Red Hat released Red Hat Network Satellite 5.5 which has OpenSCAP integrated as well, with a few features above the rival.

No matter which product do you choose, since Thursday you can enjoy documentation.

Spacewalk & OpenSCAP :: Scan Diff

What is the biggest advantage of running periodical audits? To me, it's speed by which a configuration change somewhere can be sighted and brought to my attention.

In Spacewalk, we have stepped forward and we visualize all the detected changes in a single icon. (Detailed view is available as well). For each scan we search the previous similar scan, compare them together, and show the icon of comparison. From the list of icons, it is obvious when the state of the system has changed, and the red flag flies if it deteriorates.

In the screen-shot, you see history of scans on some machine. Each line starts with icon indicating result of comparison with the previous similar scan. The icon indicates that in the newer scan there is either:

  • - no difference compared to the previous.
  • - arbitrary differences.
  • - major differences, either more failures or less passes.
  • - no comparable scan was found, no comparison was made.

Each difference is displayed only once. So even single red mark followed by dozens of green checks means that there are more failures than on the day zero. Similar icon is placed on scan's details page, near the reschedule link.

Behind the icon, there is a link to detailed comparison page. Separate tables shows diff of metadata and diff of xccdf:rule-results. Each xccdf:TestResult has separate column.

On the toolbar, there are three possible comparison modes: Full Comparison which shows all the scan items, Only Changed Items which shows only varying items, and finally Only invariant items which shows only common and equal items. The items contained only in one of the compared scans, are considered to be varying. Varying items are always highlighted with beige.

The XCCDF diff is not limited to similar scans. You can diff arbitrary scans, you need only to specify their id (so called xid) in the database.

Spacewalk & OpenSCAP :: Search

Another SCAP feature has landed in Spacewalk-nightly. It is a web page that allows users to search scan results. User can search xccdf:rules(-s), by their text fields (fulltext), by the results of evaluation, by the time, and/or by the systems.

Let's take a look how a complex query could look like. For example, I want to search systems that recently reported that they don't have boot loader locked by password. For this purpose I need the content, which checks for such configuration option and the systems my Spacewalk has scanned.

From the contents I have learned, that XCCDF rule for boot loader passwords are regarded by CCE-1818-2 identifier, so I put this down to the text field of the search form. Next, I want to see systems which are not compliant with this rule, so from the drop-down box I choose the fail result.

The third selection determines whether to search scans on all the systems, or on a subset only. Concept for managing subset of systems is called System Set Manager (SSM). And since I actually do want to search only a few systems which I've selected afore, I choose to search within the SSM. Bellow, I specify date range: since the beggining of the month up to now. And finally the last choise I leave unchanged.

The search shows that my Spacewalk keeps track of two evaluations (rule-results) of CCE-1818-2 which match given criteria and differs in the idref field (XCCDF Rule Identifier). So far so good, but I get a list of rule-results, while I would much rather get a list of systems. Hence, I change the last search choise to List of XCCDF Scans, which will search XCCDF TestResults instead. Only those TestResults are returned which contain rule-results returned by previous query.

Now, I can see that two different machines fails to have the boot loader locked. I'll fix them right after I'll have coffee.

How To Distribute SCAP Content

We know where to get an SCAP content and how to scan a machine with Spacewalk. But what is the best way to distribute the content to machines?

The answer is the RPM. (Assuming infrastucture managed by Spacewalk).

If you don't believe me, here is what I think. First of all, we have traditional distribution methods: CD, USB stick, scp, ftp, nfs, but each of them is dependent on something, either on physical acces to machine (CD, USB) or a particular software installed & running (scp, ftp, nfs, ...). Either requirement may not be met, consider a cloud or somewhat secured environment. Another option to use Spacewalk and create a script which downloads the content, and then schedule the script through Spacewalk. Well, the first thing the security department turns off is Spacewalk's remote scripting. Note that neither of these methods handles (by default) versioning, integrity, nor confidentiality.

Further, Spacewalk offers the concept of configuration files. It allows you to manage & deploy files on target machine through Spacewalk web interface. It can handle versioning and confidentiality of the content. And the process can be even automated with Spacewalk's API. However, the clients must have rhncfg* packages installed. And it might be again security requirement to not allow remote configuration management.

The RPM, on the other hand, is always present on Fedora-based systems. RPM package can be signed. RPM brings file permission (even 000 if needed). Installed RPM can be verified (rpm -V), which checks the integrity of files using SHA256 and file permissions. Moreover, with RPM you can can bring in (even custom) SELinux context. And the best is the integration of RPM in Spacewalk. For example, with Spacewalk you can schedule following actions: Install my-scap-content package. Execute scan. Verify the package. Remove it again.

I am attaching a template spec file, which can be used for an SCAP content distibution. As an example it builds from USGCB RHEL5 Desktop Baseline. scap-content-USGCB-rhel5desktop.spec.

Public SCAP Contents

As stated earlier, writing own SCAP content from scratch is rather hard. One needs to understand at least XCCDF, and often also OVAL layer. On the other hand, customization (tailoring) of existing content is easy. One needs only to review the existing and decide which checks he (or she) wants to include in his (her) security profile, and customize a few variables. There is even opensource graphical tool for tailoring, it is called scap-workbench.

There are groups publishing their XCCDF content under opensource licences. So anyone may start hacking on top of them.

  • USGCB for RHEL5 Desktop. The United States Government Configuration Baseline, It's the official SCAP content for desktops within federal agencies. It has been developed at NIST which collaborated with DoD and Red Hat, Inc. Unfortunatelly, it turned out, that the content tends to become out of date and some parts of OVAL may not fit nicely to the latest Fedoras.
  • SCAP Security Guide for RHEL6. It reassumes and extends USGCB, containing profiles for desktop, server, and ftp server. This content is under active community development.
  • SCE Community Content. This content does not use OVAL, instead it contains arbitrary scripts as checking engine. Arbitrary scripts might be no-go for some deployments, on the other hand they allow rapid development which might came handy elsewhere. The content combines rules from STIG, Aqueduct, and Sectool.
  • Content Based on Sectool and obsoleting it. It's also included in the previous.
  • OpenSCAP Content for Fedora 14. Exemplary SCAP content created for Fedora 14 and maintained by OpenSCAP community.
  • OpenSCAP Content for RHEL6.

Please let me know, in case you are aware about another group publishing their XCCDF content. I would like to have them added.

Spacewalk & OpenSCAP :: Scheduling First Security Scan

This example shows how to schedule OpenSCAP scan through Spacewalk web user interface.

Prerequisites: For this example we will need to setup server/client environment. We will use server to schedule SCAP XCCDF scan remotely on the client machine. As a client may serve any Fedora based distro (it could be e.g RHEL5), but it *could* work even with (Open)SUSE or Debian. In the previous post we have prepared a very simple SCAP content. Let's start with copying it to the client.

wget \
scp first_xccdf.xml first_oval.xml

On the server side, there will be Spacewalk with OpenSCAP support. At this point, OpenSCAP support is not yet present in any of released versions of Spacewalk, it will go to the next (post 1.7) version. Thus, we need to install Spacewalk nightly build (it may take a while). And finally, to register your client system with Spacewalk please consult documentation.

Once registered, client will appear on the Spacewalk web. As you can see on the picture, Spacewalk offers a huge variaty of management features and I suggest you to explore them.

Spacewalk Web Interface

The SCAP functionality can be found under Audit sub-tab. As the page suggests, the client machine needs to have spacewalk-oscap package installed. This package is an adapter between the /usr/bin/oscap tool and the Spacewalk. The package is available in Spacewalk-nightly client repo. You can install it with yum, or using Spacewalk webui (in that case you need to upload the package to Spacewalk first).

System Audit page.

Additionally, in case you have installed the spacewalk-oscap package using yum, you need to notify Spacewalk server that the OpenSCAP capability is available on the client. You can do it by running the /usr/bin/rhn_check command, or by waiting for rhnsd deamon to do it automatically. More info about the deamon you can find in downstream documentation: it could be either rhnsd or osad.

Now, everything is plugged in and we can schedule the OpenSCAP scan. Since we have a very simple XCCDF file (without profiles), we only to need to specify path it.

Scheduling the scan

After hitting the schedule button. the OpenSCAP action is pending and waits for client's pick-up. If you have osad daemon running, it is executed immediatelly, if you have rhnsd deamon, it is executed within next 4 hours. Alternativally, you can run it manually by rhn_checkk command.

Scan is pending.

Wondering what happens underneath? Deamon runs the rhn_check. The rhn_check reads instructions from Spacewalk and issues OpenSCAP to evaluate the given document and to store the results in a secured tempfile. Then rhn_check cherry-picks a brief resume from the results and sends it back to Spacewalk as a response. After the resume is stored Spacewalk database, user can inspect it and take actions.

Detailed results

As we can see, the client machine passed the scan. The only rule in our document no_hashes_outside_shadow is pass. Meaning, all the password hashes are safe in /etc/shadow. Yay!

A First SCAP Content

Before we start scanning we need to prepare a SCAP content. Let's start with a very simple one.

Motivation: Imagine that security specialist decided that in the organization there must not be a single machine storing password hashes in /etc/passwd. All the machines must store hashes in /etc/shadow instead. One would say that such configuration is there by default on any resonable system in 2012, but there is a slight problem with wording. Being default is not the same as being setup. For instance, consider the bug 818130, which may cause hashes stored in passwd during the kickstart. And for the record, this bug was found by OpenSCAP scan. :]

A content in this example consist of two part, XCCDF (Extensible Configuration Checklist Description Format) and OVAL (Open Vulnerability And Assesment Language). The OVAL represents the system configuration and the XCCDF builds on top of that and makes a creation of structured security checklist possible. XCCDF, is not utterly needed for our motivation example, but it's esential in more sophisticated scenarios. For example, when user want's to create single file with multiple security profiles.

For start, let's have a look at the checklist (the XCCDF document). It is XML file containing metadata for the check and for the document as whole. The top level element is <Benchmark>. Some of its items are compulsory, like document's <status> and <version> while the most of others are optional.

<Benchmark xmlns:xsi=""
    xsi:schemaLocation=" xccdf-1.1.4.xsd"
    resolved="false" xml:lang="en-US"
  <status date="2012-05-07">draft</status>
  <title>My First Benchmark</title>

All the other elements like <Rule>, <Group> of rules, and security <Profile> are enclosed within the top level <Benchmark> element. For our purposes we will need only one <Rule> element which defines that there must be no password hash in /etc/passwd.  For the <Rule> we define id, <title>, <description>, and <rationale>. And also references to another documents: the <reference> points out to further reading, the <ident> assigns the identifier from configuration enumeration system, and the <checkreferences OVAL definition.

  <Rule id="no_hashes_outside_shadow" selected="true">
    <title>Verify All Account Password Hashes are Shadowed</title>
    <description>To ensure that no password hashes are stored in
      <xhtml:code xmlns:xhtml="">/etc/passwd</xhtml:code>,
      the following command should have no output:
      <xhtml:pre xmlns:xhtml="">
      # awk -F: '($2 != "x") {print}' /etc/passwd</xhtml:pre>
      The hashes for all user account passwords should be stored in
      the file <xhtml:code xmlns:xhtml="">
      /etc/shadow</xhtml:code> and never in <xhtml:code
      which is readable by all users.
    <reference href="">
    <ident system="">CCE-14300-8</ident>
    <check xmlns:xhtml=""
      <check-content-ref href="first_oval.xml" name="oval:my_first:def:1"/>

Note that in XCCDF, there is no definition of how to make check, it is in OVAL document and referenced by oval:my_first:def:1. The OVAL document will not be discussed, firstly it is not that straight forward, and secondly in next examples we will not need to know it.

Once we have the content we can scan a machine:

# yum install openscap-utils -y
# /usr/bin/oscap xccdf eval first_xccdf.xml

Download full: first_xccdf.xml and first_oval.xml

Credits: The content in the example is based on SCAP Security Guide project on Fedora hosted.