Implementing Topic Detection with OpenAI

Topic detection is a fundamental capability in natural language processing that enables systems to automatically identify the primary subject matter within text. As organizations process increasing volumes of textual data, the ability to accurately categorize content becomes essential for content management, recommendation systems, and information retrieval.

In this guide, we'll explore how to implement robust topic detection using OpenAI's powerful language models. We'll walk through a practical implementation that achieved 88.1% accuracy in our benchmark tests, providing you with the code and insights needed to integrate this capability into your own applications.

Why Topic Detection Matters

Topic detection enables several critical capabilities in modern applications:

• More efficient content organization and retrieval
• Enhanced user experiences through better content recommendations
• Improved search functionality across large document collections
• Automated content moderation and filtering
• Data-driven insights about content trends and user interests

For AI-powered systems, accurately detecting the topic of a user's request helps route queries to specialized models or knowledge bases, providing more contextually relevant responses. This is especially valuable for enterprise deployments where understanding user intent across diverse domains, from technical support to product inquiries to policy questions, can dramatically improve response quality and user satisfaction.

Even small improvements in topic detection accuracy can translate to significant operational benefits for businesses processing thousands or millions of text documents daily.

The Dataset

Our implementation was tested on a diverse dataset comprising 2,926 text samples distributed across 14 distinct topical categories:

• Health & Medicine (235 samples)
• Education (216 samples)
• Technology (209 samples)
• Politics (207 samples)
• Food & Cooking (207 samples)
• Psychology & Self-Development (206 samples)
• Environment & Climate (206 samples)
• Entertainment (204 samples)
• Business & Entrepreneurship (204 samples)
• Travel & Tourism (203 samples)
• Science & Space (202 samples)
• Sports (201 samples)
• History (200 samples)
• Finance & Economy (185 samples)

Sample Texts from the Dataset

The table below provides a representative example from each topic category:

Implementing Topic Detection with OpenAI

Let's dive into the practical implementation of topic detection using OpenAI's models. This approach achieved an impressive 88.1% accuracy in our benchmark tests.

Getting Started with OpenAI's API

To implement topic detection with OpenAI, you'll need to:

1. Create an OpenAI account and obtain an API key from the OpenAI platform
2. Install the required dependencies:
   
   Copied!

   1pip install openai pandas tqdm python-dotenv

3. Set up your environment variables by creating a
   Copied!

   1.env

   file with your API key:
   Copied!

   1   OPENAI_API_KEY=your-api-key-here

The Implementation

Here's the complete implementation we used to achieve 88.1% accuracy with OpenAI's GPT-4 Mini model:

Copied!
1import os
2from tqdm import tqdm  # For progress bar visualization
3from typing import Any
4import pandas as pd	# For data manipulation
5import json
6from dotenv import load_dotenv  # For secure API key management
7import time  # For execution time measurement
8from openai import OpenAI
9
10# Load environment variables from .env file for secure API key storage
11load_dotenv()
12OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
13client = OpenAI(api_key=OPENAI_API_KEY)
14
15# Load the dataset containing text samples and their topic labels
16df = pd.read_csv(filepath_or_buffer="data/topic_detection.csv")
17# Extract unique topic labels to provide to the model
18labels = list(df["label"].unique())
19
20def openai_topic_detection(text: str) -> Any:
21	# Craft a detailed system prompt that instructs the model exactly how to perform classification
22	system_prompt = """
23	You are an expert on topic classification.
24	You will receive a text and you will need to classify this text depending on the topic it discusses.
25    
26	The list of topics to consider is the following:
27	{labels}
28    
29	You need to pick only one of the labels in the list and you need to use the exact same spelling.
30	Provide the output in a json with this structure:
31    
32	{{
33    	"class": "the label you picked with the exact same spelling as in the list"
34	}}
35    
36	Only answer with the json output. Do not provide any other output.
37	MAKE SURE THE LABEL HAS THE EXACT SPELLING AS IN THE LIST OF POSSIBLE LABELS
38	Example correct output:
39    
40	{{"class": "ACLASS"}}
41    
42	"""
43
44	# Call the OpenAI API with our carefully crafted prompt
45	completions = client.chat.completions.create(
46    	model="gpt-4o-mini",  # Using GPT-4o Mini for optimal accuracy/speed balance
47    	messages=[
48        	{
49            	"role": "developer",  # Using developer role for system instructions
50            	"content": system_prompt.format(labels=labels),  # Dynamically insert our topic labels
51        	},
52        	{
53            	"role": "user",  # User role contains the text to be classified
54            	"content": text,
55        	},
56    	],
57	)
58	# Parse the JSON response using eval() - in production, consider using json.loads() for safety
59	response = eval(completions.choices[0].message.content)
60	return response["class"]  # Extract just the class label from the response
61
62def main(df: pd.DataFrame):
63	# Initialize lists to store results
64	execution = list()  # Will store execution times
65	topic_detected = list()  # Will store detected topics
66    
67	# Process each text sample with a progress bar
68	for _, row in tqdm(df.iterrows(), total=len(df), desc="OpenAI Topic Detection"):
69    	text = row["text"]
70    	# Measure execution time for each classification
71    	start = time.time()
72    	topic = openai_topic_detection(text=text)
73    	end = time.time()
74    	exec = end - start
75   	 
76    	# Store results
77    	topic_detected.append(topic)
78    	execution.append(exec)
79    
80	# Add results as new columns to the dataframe
81	df["execution_time (s)"] = execution
82	df["topic_detected"] = topic_detected
83
84	# Save the results to a CSV file for analysis
85	df.to_csv(path_or_buf="data/openai_topic_detection.csv", index_label=False)
86
87if __name__ == "__main__":
88	main(df=df)  # Execute the main function when script is run directly
89

Key Implementation Details

Let's break down the key elements that make this implementation successful:

1. Precise System Prompt Engineering

The system prompt is crucial for achieving high accuracy. Our implementation:

• Establishes the model as an "expert on topic classification"
• Clearly defines the task: classify text based on topic
• Provides the exact list of possible topics dynamically
• Enforces strict output formatting as JSON
• Emphasizes the importance of exact spelling in the labels
• Includes an example of the expected output format

This careful prompt engineering ensures the model understands exactly what's expected and produces consistent, parseable results.

2. Structured JSON Output

By requesting a structured JSON response with a specific schema:

Copied!
1{
2	"class": "the label you picked"
3}
4

We ensure:

• Consistent, easily parseable responses
• No extraneous explanations or text
• Direct extraction of the classification result

The

1eval()

1response["class"]

3. Role-Based Messaging

Our implementation uses two distinct roles in the API call:

• Copied!

  1"role": "developer"

  for the system prompt that sets up the task
• Copied!

  1"role": "user"

  for the text to be classified

This separation helps the model distinguish between instructions and content to be analyzed.

4. Performance Tracking

The implementation includes built-in performance tracking:

• Execution time measurement for each classification
• Progress monitoring with tqdm
• Storage of results for later analysis

This allowed us to accurately measure the 0.65 second average processing time reported in our benchmark.

Adapting the Implementation for Your Needs

To use this code for your own topic detection needs:

1. Prepare your dataset: Ensure your data has a "text" column containing the content to classify and a "label" column with the ground truth (if evaluating performance).

2. Customize the topic list: If you have a fixed set of topics, you can hardcode them instead of extracting them from the dataset:

Copied!
1   labels = ["Technology", "Health & Medicine", "Finance & Economy", "Politics", ...]
2

3. Adjust the model: While we used "gpt-4o-mini" for our benchmark, you can experiment with other OpenAI models:

Copied!
1   model="gpt-4" # For highest accuracy but higher cost
2   model="gpt-3.5-turbo" # For lower cost with reasonable accuracy
3

Performance Results

In our benchmark tests, this OpenAI implementation achieved:

• 88.1% topic classification accuracy - correctly identifying topics in nearly 9 out of 10 text samples
• 0.650 seconds average processing time per text sample

This represents an excellent balance of accuracy and speed for most practical applications.

Conclusion

Implementing topic detection with OpenAI's models provides a powerful way to automatically categorize text content with high accuracy. The implementation we've shared achieved 88.1% accuracy across a diverse dataset of topics, making it suitable for a wide range of applications from content management to user intent classification.

By following the approach outlined in this guide, you can quickly integrate sophisticated topic detection capabilities into your own applications, leveraging the power of OpenAI's language models to understand and categorize textual content.

Whether you're building a content recommendation system, enhancing search functionality, or developing an AI assistant that needs to understand user queries, this implementation provides a solid foundation for accurate topic detection.