Merge pull request #18 from Neverbolt/unify

Adds documentation for use cases and configurable

Author: andreashappe

README.md

@@ -132,6 +132,10 @@ wintermute.py: error: the following arguments are required: {linux_privesc,windo
 $ python wintermute.py linux_privesc --enable_explanation true --enable_update_state true
 ~~~
 
+## Contribution
+
+If you want to contribute additional use-cases, please take a look at [docs/use_case.md](docs/use_case.md) for a start on how to extend this codebase.
+
 # Disclaimers
 
 Please note and accept all of them.

docs/configurable.md

@@ -0,0 +1,88 @@
+# Configurable
+
+Marking a class as `@configurable` allows for the class to be configured via command line arguments or environment variables.
+
+This is done by analyzing the parameters to the class' `__init__` method and its `__dataclass_fields__` attribute if it is a `@dataclass`.  
+As having a `@configurable` also be a `@dataclass` makes it easier to extend it, it is usually recommended to define a configurable as a `@dataclass`.
+Furthermore, using a dataclass allows more natural use of the `parameter()` definition.
+
+All [use-cases](use_case.md) are automatically configurable.
+
+## Parameter Definition
+
+Parameters can either be defined using type hints and default values, or by using the `parameter()` method.
+
+```python
+from dataclasses import dataclass
+from utils.configurable import configurable, parameter
+
+
+@configurable("inner-example", "Inner Example Configurable for documentation")
+@dataclass
+class InnerConfigurableExample:
+    text_value: str
+    
+
+@configurable("example", "Example Configurable for documentation")
+@dataclass
+class ConfigurableExample:
+    inner_configurable: InnerConfigurableExample
+    text_value: str
+    number_value_with_description: int = parameter(desc="This is a number value", default=42)
+    number_value_without_description: int = 43
+```
+
+As can be seen, the `parameter` method allows additionally setting a description for the parameter, while returning a `dataclasses.Field` to allow interoperability with existing tools.
+
+The type of a configurable parameter may only be a primitive type (`int`, `str`, `bool`) or another configurable.
+
+## Usage
+
+When a class is marked as `@configurable`, it can be configured via command line arguments or environment variables.  
+The name of the parameter is automatically built from the field name (in the case of the example to be `text_value`, `number_with_description` and `number_value_without_description`).
+
+If a configurable has other configurable fields as parameters, they can be recursively configured, the name of the parameter is built from the field name and the field name of the inner configurable (here `inner_configurable.text_value`).
+
+These parameters are looked up in the following order:
+
+1. Command line arguments
+2. Environment variables (with `.` being replaced with `_`)
+3. .env file
+4. Default values
+
+When you have a simple use case as follows:
+
+```python
+from dataclasses import dataclass
+from usecases import use_case, UseCase
+
+@use_case("example", "Example Use Case")
+@dataclass
+class ExampleUseCase(UseCase):
+    conf: ConfigurableExample
+    
+    def run(self):
+        print(self.conf)
+```
+
+You can configure the `ConfigurableExample` class as follows:
+
+```bash
+echo "conf.text_value = 'Hello World'" > .env
+export CONF_NUMBER_VALUE_WITH_DESCRIPTION=120
+export CONF_INNER_CONFIGURABLE_TEXT_VALUE="Inner Hello World"
+
+python3 wintermute.py example --conf.inner_configurable.text_value "Inner Hello World Overwrite"
+```
+
+This results in 
+
+```
+ConfigurableExample(
+    inner_configurable=InnerConfigurableExample(text_value='Inner Hello World Overwrite'),
+    text_value='Hello World',
+    number_value_with_description=120,
+    number_value_without_description=43
+)
+```
+

docs/use_case.md

@@ -0,0 +1,28 @@
+# Use Cases
+
+Wintermute consists of different use-cases (classes extending `UseCase`, being annotated with `@use_case` and being imported somewhere from the main `wintermute.py` file), which can be run individually.
+
+The `@use_case` annotation takes a name and description as arguments, which are then used for the sub-commands in the command line interface.
+
+When building a use-case, the `run` method must be implemented, which is called after calling the (optional) `init` method (note that this is not the `__init__` method).  
+The `run` method should contain the main logic of the use-case, though it is recommended to split the logic into smaller methods that are called from `run` for better readability (see the code for [`RoundBasedUseCase`](#round-based-use-case) for an example).
+
+A use-case is automatically a `configurable`, which means, that all parameters of its `__init__` function (or fields for dataclasses) can be set via command line / environment parameters. For more information read the [configurable](configurable.md) documentation.  
+It is recommended to define a use case to be a `@dataclass`, so that all parameters are directly visible, and the use-case can be easily extended.
+
+## General Use Cases
+
+Usually a use case follows the pattern, that it has a connections to the log database, a LLM and a system with which it is interacting.
+
+The LLM should be defined as closely as necessary for the use case, as prompt templates are dependent on the LLM in use.  
+If you don't yet want to specify eg. `GPT4Turbo`, you can use `llm: OpenAIConnection`, and dynamically specify the LLM to be used in the parameters `llm.model` and `llm.context_size`.
+
+In addition to that, arbitrary parameters and flags can be defined, with which to control the use-case. For consistency reasons please take a look if similar parameters are used in other use cases, and try to have them act accordingly.
+
+When interacting with a LLM, the prompt and output should always be logged `add_log_query`, `add_log_analyze_response`, `add_log_update_state` or alike, to record all interactions.
+
+## Round Based Use Case
+
+The `RoundBasedUseCase` is an abstract base class for use-cases that are based on rounds where the LLM is called with a certain input and the result is evaluated using different capabilities.
+
+An implementation needs to implement the `perform_round` method, which is called for each round. It can also optionally implement the `setup` and `teardown` methods, which are called before and after the rounds, respectively.