Cybersecurity Dataset Construction via AI Agents & SAST LLM Fine-Tuning

This project aims to design a sovereign source code security audit model (Static Application Security Testing). By leveraging pipelines of agents to collect, clean, and distill vulnerability data, a high-quality dataset of over 20,000 samples was constructed. An open-source model (Qwen 3.6 27B) was then fine-tuned using QLoRA (FP8) on this dataset, offering a sovereign, low-cost solution that protects sensitive code.

Context

This work was completed as part of my 6-month final year internship at Orange, within the DEFY security team.

Faced with increasingly frequent and sophisticated cyberattacks, notably powered by AI, switching to AI for defense has become essential. However, traditional solutions present major limitations:

• Classic SAST Tools: They display an extremely high false positive rate, overloading security teams with irrelevant alerts.
• Proprietary Models/Third-Party APIs: Commercial solutions are expensive, their rates are constantly rising, and sending source code to non-sovereign third-party servers prevents auditing sensitive internal applications.

The objective of the mission was therefore to design a sovereign, local, and high-performance alternative: building a cybersecurity dataset to train an open-source model that can be deployed on-premises at Orange, ensuring data confidentiality and controlled operating costs.

Data

Constructing a quality training dataset was the first major challenge, as public databases lack actionable code context.

• Vulnerability Databases: Leveraging sources cataloging CVEs such as NVD, Github Advisory Database, Snyk, and OSV.
• Incomplete Databases: Public CVE databases often only provide a brief text explanation and a link to the fixing commit. They do not directly contain the affected code segments, making direct training impossible.
• Internal Reports: Access to audit reports of Orange’s internal applications and Bug Bounty program reports.

Methodology

The development pipeline was structured around these key stages, ranging from automated collection to real-world evaluation.

1. Scraping and Extraction by AI Agents

To retrieve the vulnerable code and its context associated with public CVEs, an agentic pipeline based on the LangGraph framework and connected to specialized tools was developed:

• Serena MCP: Allows the agent to dynamically explore Git repositories and extract business context.
• Tree-Sitter: Generates Abstract Syntax Trees (AST) to build a graph of methods linking callers and callees.
• LLM Routing: The agent relied on Claude Haiku 4.5 for standard exploration tasks, with automatic escalation to Sonnet 4.6 in case of blockage or failure, all structured with strict JSON output formatting.
• Result: Collected nearly 25,000 vulnerability samples containing source code, business context, the applied patch, and the associated description.

2. Noise Filtering

Raw cybersecurity datasets contain a massive amount of noise (test code, empty commits, wrong associations). To address this:

• Semantic Filter: An initial semantic sort eliminated samples without code or clearly off-topic.
• Agentic Evaluation Pipeline: A second evaluation pipeline (powered by Sonnet 4.6) analyzed each remaining sample in depth to validate the real link between the extracted code and the described vulnerability, discarding false positives.

3. Knowledge Distillation & Chain-of-Thought (CoT)

To teach the model to reason like a security analyst, raw, highly heterogeneous vulnerability descriptions were discarded.

• CoT Generation: A “teacher” LLM (Sonnet 4.6) was utilized to write detailed explanations describing the complete flow of sensitive user data (from sources to sinks) and the exploit path.
• Patch Explanations & Hard Negatives: Detailed analyses of the fixes (e.g., implementing a sanitizer) were also written to explain why the patched code is secure. The dataset was split equally (50% vulnerable with exploit explanation / 50% secure with fix explanation).
• Result: A final high-quality, balanced dataset of over 20,000 samples covering all web languages and most CWEs.

4. Fine-Tuning with QLoRA

To adapt the model to the task while respecting hardware budget constraints:

• Model Selection: After benchmarking cybersecurity models in the <35B parameter range, Qwen 3.6 27B was selected. Its dense version proved more effective for the complex logical reasoning required in security than the Mixture of Experts version (Qwen 3.6 35B A3B).
• Native FP8 Quantization: To halve the memory footprint with negligible performance loss, the FP8 format was prioritized. Rather than quantizing the model customly using Unsloth, the official FP8 weights from the creator on Hugging Face were used.
• Hardware & Training: Since this native FP8 setup requires a GPU that supports FP8 hardware acceleration, the model was trained using the Transformers library on an NVIDIA H100 GPU (80 GB) on GCP.

5. Deployment and LoRA Architecture

The model was deployed in production using vLLM on a cluster of two NVIDIA L4 GPUs (24 GB VRAM each). vLLM allows running the base model while dynamically loading the fine-tuned LoRA adapter. This flexible architecture allows teams to infer with or without the security adapter on demand for comparative evaluation of the fine-tuned model against the baseline model.

6. Evaluation Protocol

Validation of this SAST LLM is based on three steps:

1. Academic Evaluation: Testing on 10% of the dataset excluded from the training phase (validation set).
2. Real-world Use Case: Auditing an application previously tested by team pentesters to verify if the model successfully identifies the same vulnerabilities without generating false positives.

Results

The project successfully delivered a powerful sovereign model, highly specialized in static application security testing (SAST), deployable locally to guarantee absolute confidentiality of the source code.