AI News Blog System.

Description

AI News Blog System is an automated news portal split into two decoupled components that talk to each other over a REST API. A Laravel application owns the database, serves the public frontend and admin panel, and exposes the API; a separate Python worker runs on a schedule to scrape sources, rewrite content with a language model, and push finished articles back into the portal.

The design deliberately keeps a single source of truth: Laravel is the only system that writes to the database, so validation, slugs, events, and caching all stay centralized while the worker stays a stateless batch job.

Background

Manually curating and publishing a steady stream of news is time-consuming, and letting a scraper write straight into the database tends to scatter business logic and create data-integrity problems. This project was built to explore an end-to-end content automation pipeline while keeping a clean architectural boundary between the data owner and the automation that feeds it.

Goal

The goal was to build a config-driven, hourly news pipeline where new sources can be added from an admin panel without touching worker code, where duplicate articles are prevented by design, and where a failing LLM never publishes broken content. Two systems were meant to run on separate servers, communicating only through an authenticated HTTPS API.

Features

• REST API (Laravel Sanctum tokens with granular abilities) as the only write path into the database.
• Config-driven scraping: sources, keywords, and CSS selectors are managed from the admin panel and pulled by the worker via API.
• RSS-first ingestion, with HTML scraping as a fallback for sites without a feed; main article content is extracted with trafilatura and normalized to clean Markdown.
• Keyword filtering and content deduplication via a unique source_hash (idempotent uploads — reruns are safe).
• LLM rewriting using free providers (Groq / Gemini / OpenRouter / local Ollama) with fallback between providers; on failure the article is saved as a draft instead of being published.
• Auto-publish with a publish event that triggers Telegram notifications.
• WordPress-style admin panel: dashboard, articles, categories, scrape sources, comment moderation, and a settings page that overrides config at runtime.
• Public frontend: hero and grids, category and sub-category navigation, related news, social share buttons, ad slots, and comments with moderation.
• Sidebar widgets: weather (Open-Meteo), prayer times (Aladhan), trending, most popular, recent comments, and affiliates, each hiding itself gracefully on API failure.

Technologies

• Laravel 13 on PHP 8.3+, with Laravel Sanctum and Blade templates
• MySQL for storage
• Python worker using requests, feedparser, BeautifulSoup, and trafilatura
• Free LLM providers (Groq, Gemini, OpenRouter, Ollama) with a fallback strategy

How to Run

Run the portal with composer install, configure .env for MySQL, run migrations, and start it with php artisan serve. Generate a worker token with php artisan worker:token <name> and manage scrape sources from the admin panel. For the worker, create a Python virtual environment, install requirements.txt, set API_BASE_URL and API_TOKEN (plus optional LLM keys) in .env, then run python main.py — typically scheduled hourly via cron.

Technical Decisions

The central decision was making Laravel the sole database owner so that all integrity, validation, and caching logic stays in one place, with Python communicating only through the API. RSS was preferred over HTML scraping for stability and legality, and configuration was pushed into the database so new sources need no code changes. Idempotency was built in via a unique source_hash, and a deliberate “emergency brake” saves articles as drafts whenever the LLM fails rather than risking auto-publishing broken text.

What I Learned

This project reinforced how valuable a strict architectural boundary is: keeping the scraper stateless and the API as the only write path made the whole system easier to reason about and safe to rerun. It was also a practical exercise in orchestrating multiple free LLM providers with fallback, extracting clean article content from messy HTML, and treating source attribution as a minimal but important mitigation for LLM hallucination.

Notes

The repository is public. A screenshot or live preview may be added later. Because articles are LLM-rewritten, source URLs are always stored as attribution, and the pipeline respects robots.txt on source sites; the LLM can still misstate facts, so attribution is a mitigation rather than a guarantee.