ProjectGen is a multi-agent framework that decomposes projects into architecture design, skeleton generation, and code filling stages with iterative refinement and memory-based context management.

The core implementation resides in the src/ directory, which contains three major components: the multi-agent system, the memory management module, and a set of workflow and utility scripts. The directory structure is shown below:

src/
├── agents/
│   ├── architecture_agent.py
│   ├── arch_judge_agent.py
│   ├── skeleton_agent.py
│   ├── skeleton_judge_agent.py
│   ├── code_agent.py
│   ├── code_judge_agent.py
│   └── test.py
│
├── memory_manager/
│   ├── arch_memory.py
│   ├── skeleton_memory.py
│   ├── code_memory.py
│   └── __init__.py
│
├── build_dependency_graph.py
├── extract_api.py
├── logger.py
├── main.py
├── prompts.py
├── utils.py
└── workflow.py

• agents/: This directory contains the core agents used in the three-stage generation workflow. Each stage consists of a generation agent and a judging agent, forming a generate–evaluate–refine loop.

• memory_manager/: The memory modules preserve intra-stage semantic information, enabling agents to efficiently access relevant context from previous iterations.

• build_dependency_graph.py: Constructs file-level dependency graphs to support ordering.

• extract_api.py: Extracts function signatures from generated code to support iteration.

• logger.py: A unified logging module for debugging, tracing agent outputs, and monitoring workflow execution.

• main.py: The main entry point of the system.

• prompts.py: Contains prompt templates used by all agents.

• utils.py: Provides general-purpose utility functions.

• workflow.py: Defines the overall multi-agent generation workflow.

conda create -n projectgen python=3.9
conda activate projectgen
pip install -r requirements.txt

Open src/utils.py and fill in your OpenAI API key:

os.environ["OPENAI_API_KEY"] = "your_openai_api_key_here"

Navigate to the src directory and run the main script:

cd src
python main.py --dataset=CodeProjectEval

The outputs will be stored in the CodeProjectEval_outputs/ folder.

To better reflect real-world project scenarios and support evaluation through executable test cases, we construct a new project-level code generation dataset, CodeProjectEval, which consists of 18 Python repositories covering a wide range of topics.

Each repository is supplemented with:

• docs/PRD.md: provides detailed descriptions of a software system’s functional and non-functional requirements, guiding subsequent design and development.
• docs/UML_pyreverse.md: UML class diagram and package diagram generated by Pyreverse.
• docs/architecture_design.md: the directory tree of the repository and descriptions for each source file, accompanied by summaries of the classes and functions they contain.
• check_tests/: to provide initial verification during the code generation process.
• unit_tests/: executed upon completion of code generation to evaluate the overall quality and functional correctness of the generated projects.

• Similarity-based Evaluation (SketchBLEU)

  We adopt SketchBLEU, a similarity-based metric originally introduced in the CodeS framework. This metric evaluates the structural similarity between the generated code and the reference implementation. To calculate SketchBLEU:

  cd datasets/evaluation
  python calc_sketchbleu.py
  

• Unit Test–based Evaluation

  For functional correctness evaluation, each repository contains a set of unit tests. To run the unit tests:

  cd datasets/evaluation
  python calc_passrate.py