Generative AI Pilot Prompt Analysis

The AI Lab of the municipality of Amsterdam is experimenting with generative AI to advise on its practical use within municipalities. Earlier, we compared different large language models (LLMs). More recently, a pilot was conducted where 150 civil servants used a GPT-based chatbot tool for four weeks and the AI Lab analyzed the prompts received by the tool during this pilot phase. The study focused on analyzing user behavior, bias, and factuality. Based on the analysis findings, different risk mitigation strategies were proposed in the full report.

This repository contains the code that was used for the semi-automatic analysis of user bahavior and bias in the prompts.

Contents

• data: Sample data for demo purposes. Disclaimer: this data does not come from the pilot, but was manually curated to resemble the prompts from the pilot.
• notebooks: The notebooks for prompt and bias analysis
• report: The Dutch and English versions of the corresponding report.

Installation

1. Clone this repository:

git clone https://github.com/Amsterdam-AI-Team/gen-ai-pilot-prompt-analysis.git

2. Install all dependencies:

pip install -r requirements.txt

The code has been tested with Python 3.9 on Linux.

Configuration

Contributing

Feel free to help out! Open an issue, submit a PR or contact us.

Acknowledgements

This repository was created by Amsterdam Intelligence for the City of Amsterdam.

License

This project is licensed under the terms of the European Union Public License 1.2 (EUPL-1.2).