Developing a rule

GitHub Use Case

Note

To enhance your understanding of this readme, we will provide examples throughout using a GitHub use case. The use case is a very simple control that enforces protection for all branches in your GitHub repository. This control can be automated by accessing your GitHub repository, listing all branches, and verifying whether protection is enabled for each of them.

The development step consists of 4 sub-steps as demonstrated in the GIF below.

1.1 Creating a credential type

As illustrated in the GIF above, credentials play a crucial role in enabling the task code to access applications. Therefore, let's begin by defining the specific type of credentials required for this purpose.

1.1.1 cowctl init credential

Initialize a new credential type (config YAML file) with the init credential command. You'll be prompted for a name and version.

You can then locate the credential YAML file at catalog/gobalcatalog/yamlfiles/credentials. Customize the attributes in it to match your credential type, like connecting to GitHub with a Personal Access Token (1 attribute) or username/password (2 attributes).

Here is an example from the GitHub use case.

A few attributes are omitted here for simplicity.

apiVersion: v1alpha1
kind: credentialType
meta:
  name: GithubTokenCred 
  displayName:  Github Token Credential
  version: 1.1.1 # optional. 
spec:
 extends: [] # optional.
 attributes:
 - name: personalAccessToken
   displayName: Personal Access Token
   secret: true # optional. boolean: true | false. If true, this attribute value should go to the vault
   required: true
   multiSelect: false
   dataType: STRING
   allowedValues: [] # optional
   defaultValue: # optional

1.1.2 cowctl create credential

After defining the attributes in the credential YAML file, run create credential to validate it and copy it to catalog/globalcatalog/declaratives/credentials, where the subsequent cowctl commands will reference it.

You will need to choose the YAML file for the credentials that you've just created.

1.2 Creating an application type

An application type is essentially a class in Go or Python, where you can implement these:

• a validation method to verify the accessibility of the application using the provided credentials.
• methods (for accessing the application or its resources) that can subsequently be used within the task code.

1.2.1 cowctl init application

Use the init application command to initialize the application type (YAML file template). Following the name prompt, you'll be prompted to choose a credential, along with its version, to bind with the application (for access). If you've created multiple credentials, you'll be provided with an option to bind additional credentials to the application.

The init application command will create a file in catalog/globalcatalog/yamlfiles/applications where important fields will be auto filled.

Here is an example from the GitHub use case.

A few values are omitted here for simplicity.

apiVersion: v1alpha1
kind: applicationClass
meta:
  name: GitHubApp
  displayName: GitHubApp # Display name
  labels: # required. The rule orchestrator selects the INSTANCE of the APPLICATION CLASS based on the labels described here
    environment: [logical] # Application group type
    execlevel: [app] # Application group level is 'app' (supported types are 'app' and 'server')
    appType: [githubapp]
  annotations: # optional. These are user defined labels for reporting purposes
    appType: [githubapp]
  version: 1.1.1 # semver
spec:
  url: http://localhost.com
  port: # port
  credentialTypes:
  - name: GithubTokenCred
    version: 1.1.1

1.2.2 cowctl create application

You can verify the details and modify the application YAML file as needed or continue to create application which will create the actual application package and the classes (including that of the credential type). You will be asked to select the application type (YAML file). Please select the one that you just created.

Once the create application is completed, a new package and a set of classes will be created in catalog/appconnections/go/ for Golang and catalog/appconnections/python/appconnections for Python. Package name shall be the same as the application name.

1.3 Initializing a rule

Now that we have created the application type and its credentials, it's time to initialize a rule with its associated tasks. Use the init rule command which will prompt you to provide:

• Enter a name for the rule.
• Specify the number of tasks you want to include in the rule.
• Provide the name and programming language (Go/Python) for each task.
• Select the application type to bind with each task (This is required to import the application class package into the task and auto-fill the code related to the application package.)
• Choose the primary application type to bind with the rule (This is necessary when different tasks use different application types)

After entering all the necessary inputs, the console will display the paths where both the Rule and the generated Tasks can be found.

• The rule Folder will contain these:
  • rule.yaml: This essentially orchestrates the flow and dependencies between tasks, facilitating a structured execution of the overall process. Within rule.yaml, you can expect to find details such as the names of the tasks, the specified execution order for the tasks, and the I/O wiring.
  • inputs.yaml: This is explained in a section further down below.

Wiring the tasks - Input & Output

The next step is to configure the Inputs and Outputs. A rule shall accept a set of inputs and shall produce a set of outputs. These inputs can be fed to any of its tasks as task inputs. Similarly, outputs of any of its tasks can be extracted out as rule outputs. Additionally, you can wire output of one task as input for another task in the sequence. In the example shown below, the rule has two tasks (Task1 and Task2), where Task1 accepts 1 input and produces two outputs, and Task2 accepts 2 inputs and produces 1 output. An output (t1_out2) of Task1 is fed as input (t2_in1) to Task2.

The above can be achieved by updating the "ioMap" of your rule in a file named rule.yaml in the rule folder.

For the example given in the diagram above, the refmaps arrays shall look like the one below . In essence, for each arrow shown in the diagram above, there should be an object in the ioMap array. Here is how to read each object. Let's take the 1st object. It is about mapping the Input R_In1 of Rule (aliasref is *) to Task1 as its input called T1_In1. Alias "*" represents Rule level. Field type specifies whether the value is an input or an output.

  ioMap:
      - 'Task1.Input.T1_In1:=*.Input.R_In1'
      - 'Task2.Input.T2_In2:=*.Input.R_In2'
      - 'Task2.Input.T2_In1:=Task1.Output.T1_Out2'
      - '*.Output.R_Out1:=Task1.Output.T1_Out1'
      - '*.Output.R_Out2:=Task2.Output.T2_Out1'

If you wish to calculate the Compliance Status and Compliance Percentage within any of your tasks, please include two additional objects in the refmaps, as demonstrated below. It's important to note that the variable names (varname) must be set to 'ComplianceStatus_' and 'CompliancePCT_'. If these values are returned by multiple tasks, only the data from the last task will be exposed at the rule level.

    - '*.Output.ComplianceStatus_:=Task2.Output.ComplianceStatus_'
    - '*.Output.CompliancePCT_:=Task2.Output.CompliancePCT_'

Here is an example from the GitHub use case.

In this context, the input 'GitHubRepoName' is forwarded to Task t1. Subsequently, Task t1 processes this information and transmits 'RepoInfo' to Task t2. Task t2 generates output data comprising Compliance Percentage, Compliance Status, and Branches Summary Data. The output produced by Task t2 will be referred to as the rule's output.

  ioMap:
      - 't1.Input.GitHubRepoName:=*.Input.GitHubRepoName'
      - 't2.Input.RepoInfo:=t1.Output.RepoInfo'
      - '*.Output.CompliancePCT_:=t2.Output.CompliancePCT_'
      - '*.Output.ComplianceStatus_:=t2.Output.ComplianceStatus_'
      - '*.Output.BranchesSummaryData:=t2.Output.BranchesSummaryData'

inputs.yaml

All inputs required to execute a rule should be specified in inputs.yaml which contains:

• userObject: Many rules necessitate an application's ability to retrieve and process data. To facilitate this, you must furnish the credentials to grant the rule access to these applications, in userObject.
• userInputs which contains the user input values required for the rule and task execution.

The inputs.yaml file will be automatically generated upon the initialization of a rule. Depending on the chosen application class, the userObject will have its credentials structure auto-populated.

Important Note: To keep critical credential values away from code, we recommend to use etc/.credentials.env (which is git ignored) as given below.

GITHUB_PERSONAL_ACCESS_TOKEN=github_pat_....

Then, use this env key in the inputs.yaml.

Here is an example from the GitHub use case.

userObject:
    apps:
      - name: GitHubApp
        appURL: https://api.github.com
        appPort: "0"
        appTags:
            appType:
            - githubapp
            environment:
            - logical
            execlevel:
            - app
        userDefinedCredentials:
            GithubTokenCred:
                personalAccessToken: "$GITHUB_PERSONAL_ACCESS_TOKEN"
userInputs:
    BucketName: demo
    GitHubRepoName: "PyGithub/PyGithub"

Below is a class diagram that explains the relationship among the Task, the App and the inputs.

1.4 Implementing the business logic

Both your task and application code will need to utilize the various auto-generated PolicyCow classes. To facilitate seamless integration in your IDE, please execute the command sh install_cow_packages.sh (in Mac or Ubuntu) or ./windows_setup/install_cow_packages.ps1 (in Windows). You can find this file in the main folder of PolicyCow.

In your task, the typical approach is to access your application to retrieve data and subsequently process it. To maintain a clear separation of concerns, it is advisable to refrain from writing code in your task that directly pertains to application access. Therefore, let's begin by implementing the application-specific methods within the application class.

Note: If you require any third-party libraries, you are free to install and incorporate them into your code.

Please adhere to the relevant set of instructions, depending on your chosen programming language for task development.

Implementing the application methods

Note that the package name shall be the same as the application name.

Python instructions

You need to modify catalog/appconnections/python/appconnections/<package name>/<package name>.py, which includes several default classes, namely {{ApplicationClassName}}, UserDefinedCredentials, and {{CredentialName}}(s), as depicted in the class diagram above. Each of these classes is furnished with 'from_dict' and 'to_dict' methods. These methods streamline the object initialization process by allowing the loading of values from any YAML file.

The pre-populated validate method shall look like the below. You can add custom logic to validate the application's access by utilizing the provided credentials.

def validate(self)->bool and dict:
    return True, None

Within any of your custom methods, you can access the application attributes such URL, port, credentials, etc, as mentioned below, to retrieve data from the application:

class YourApp:
    ...
    def your_custom_method(self) -> dict:
        # self.app_url
        # self.app_port
        # self.user_defined_credentials.<credential_obj>.<credential_attribute>
    ...

Here is an example from the GitHub use case, where a custom method get_branches is implemented.

class GitHubApp:
    app_url: str
    app_port: int
    user_defined_credentials: UserDefinedCredentials
    ...
    def get_branches(self, repo_name) -> dict:
        auth = Auth.Token(self.user_defined_credentials.github_token_cred.personal_access_token)
        github_obj = Github(auth=auth, base_url=self.app_url)
        repo = github_obj.get_repo(repo_name)
        return repo.get_branches()
    ...

Now, you can invoke the get_branches method from your task code. Further details on this are explained in the following section.

Go instructions

You need to modify catalog/appconnections/go/<package name>/<package name>.go file, which includes struct definitions for {{ApplicationClassName}}, UserDefinedCredentials, and structs for {{CredentialsName}}(s). UserDefinedCredentials is basically a wrapper for the credential classes.

By default, a Validate method is provided in the following format:

func (thisObj *{{ApplicationName}}) Validate() (bool, error) {
    return true, nil
}

Implementing the tasks

Python instructions

As explained in the class diagram above, a class called Task in the task.py file within your task folder shall have an execute method where you can house the task's business logic.

You can access the user defined credentials details under userObject key and user input values under userInputs key in inputs.yaml using self.task_inputs.user_object.app.user_defined_credentials and using self.task_inputs.user_inputs.

The application package chosen during the rule initialization will already be automatically imported into the task.py file. To instantiate the application class with the userDefinedCredentials from the inputs.yaml file, you can use the <packageName>.UserDefinedCredentials.from_dict() method and pass the self.task_inputs.user_object.app.user_defined_credentials in it. It is because the App's constructor expects UserDefinedCredentials object and not a dict.

Here is an example from the GitHub use case, where the app's method get_branches is invoked.

class Task(cards.AbstractTask):
    ...
    def execute(self) -> dict:
        app = githubapp.GitHubApp(
            app_url = self.task_inputs.user_object.app.application_url,
            user_defined_credentials = githubapp.UserDefinedCredentials.from_dict(self.task_inputs.user_object.app.user_defined_credentials)
        )
        self.task_inputs.user_inputs.get("GitHubRepoName")
        branches = app.get_branches(repo_name)
        ...
        # Assess branches for its protection and create branches_summary dict
        ...
        file_hash, file_path, error = self.upload_file(
            file_name="branches-summary",
            file_content=pandas.DataFrame(branches_summary).to_json(orient='records'), 
            content_type=".json"
        )
        ...
    ...

Note: You can upload and download files to minio using self.upload_file(...) and self.download_file(...) methods, as shown above. Files uploaded to Minio can be configured as "Evidence" files in ComplianceCow.

Go instructions

The task folder shall have four Go files, inputs.yaml file and a go.mod file. In task_service.go a function bearing the same name as the task is where you'd craft the core business logic for that task.

func (inst *TaskInstance) TaskName(inputs *UserInputs, outputs *Outputs) (err error){}

In the task_service_structs.go file, populate the userInputs struct with values that mirror those specified in the inputs.yaml file. Additionally, ensure that the struct fields have YAML tags that match the letter case of the keys in the YAML file.

type UserInputs struct {
    R_In1 string `yaml:"R_In1"`
    R_In2 string `yaml:"R_In2"`
}

You can import the applicationClass package to the task_service.go, if required. The package will be present in task_serverStrcts.go file in the below format.

import (
    {{packageName}} "appconnections/{{packageName}}"
)

Once these are done, run go mod tidy to get all the packages required. In case of any errors in accessing appconnections or cowlibrary, modify the go.mod file relative path in such a way that it will be able to access the catalog/appconnections/go path.

replace cowlibrary => ../../../../src/cowlibrary
replace appconnections => ../../../appconnections/go

Now you will be able to access the app defined under userObject and userInputs keys in inputs.yaml using inst.SystemInputs.UserObject and inst.UserInputs.

You can initiate the application class, as mentioned below, to validate the credentials and to use other methods that is already implemented in the application package.

    varName := &packageName.ClassName{UserDefinedCredentials: &inst.SystemInputs.UserObject.App.UserDefinedCredentials}

Now you can continue with your implementation of the task.

Note: You can import appconnections/minio to upload and download files. Please check catalog/appconnections/go/minio to see the available methods.