Overview
A generative-AI platform that monitors terrorism, antisemitism, and extremist content across multiple online platforms at scale. The system ingests, processes, and analyzes content from Telegram channels, 4chan, 8kun, and other sources — using LLM-based pipelines to surface actionable intelligence through automated summarization, semantic search, and periodic briefing reports.
Architecture
Six-layer architecture: Data Sources → Ingestion → Staging & Filtering → Processing → Storage → Intelligence
Coverage
- Telegram: Over 1,000 channels, categorized by geography and subject
- 4chan, 8kun: Active ingestion and analysis
- Expanding: Additional sources being integrated
Tech Stack
Python / FastAPI LLM APIs RAG Embeddings Vector DB Hierarchical Summarization React
📍 GCP Cloud Run 📍 Cloud SQL 📍 BigQuery 📍 GCP VM 📍 Firebase 🔐 IAP + OAuth2 Multi-Tenant
Deployment
| Layer | Infrastructure | Runtime |
|---|---|---|
| Ingestion | GCP Cloud Run (always-on) | 24×7 real-time |
| Staging & Filtering | GCP Cloud Run (always-on) | Real-time |
| Processing | GCP Cloud Run (on-schedule) | Periodic batch |
| Storage | Cloud SQL + BigQuery | GCP managed |
| Intelligence BE | GCP VM | Always-on |
| Frontend | Firebase | Always-on |
| Authentication | IAP (admins) + OAuth2 multi-tenant | — |
Scale & Capabilities
- Tens of thousands of posts ingested per channel
- Over 1,000 Telegram channels mined at near real-time
- Cross-platform correlation across Telegram, 4chan, 8kun, etc.
- Hierarchical summarization: individual posts → thread-level themes → cross-platform intelligence
- RAG-based conversational Q&A over indexed content
- Real-time alert rules with threshold-based notifications
- Periodic briefing with intelligent report generation
- Custom categorization, entity extraction, and sentiment analysis
Development Approach
The entire system was designed, architected, and deployed by a single developer leveraging AI-assisted coding tools throughout the workflow — from rapid prototyping through production DevOps. This approach enabled iterative architecture refinement with real-time AI pair programming, automated testing and deployment pipelines, and full ownership of the entire lifecycle.