Python News Aggregator Web App Project

In today’s fast‑paced digital world, staying on top of the latest headlines can feel like chasing a moving target. A Python‑powered news aggregator web app solves that problem by pulling stories from multiple sources, normalizing the data, and presenting it in a clean, customizable interface. Whether you’re a beginner looking for a hands‑on project or a seasoned developer aiming to showcase your full‑stack skills, building a news aggregator with Python offers a perfect blend of API integration, data processing, and web development. In this guide we’ll walk through everything you need to know to plan, code, and launch a robust news aggregator that not only works flawlessly but also ranks well in search engines.

What Is a News Aggregator and Why Build One?

A news aggregator is a platform that collects articles, blog posts, and other content from various sources—RSS feeds, public APIs, or web‑scraped pages—and displays them in a single, unified feed. The main benefits of creating your own aggregator include:

• Personalization: Tailor the content to specific topics, languages, or regions.
• Learning Opportunity: Gain hands‑on experience with HTTP requests, JSON handling, and database design.
• Portfolio Boost: Demonstrate full‑stack capabilities to potential employers or clients.
• Monetization Potential: Add ads, premium features, or affiliate links for revenue.

Why Choose Python for This Project?

Python’s readability, extensive libraries, and vibrant community make it an ideal choice for a news aggregator. Here are the top reasons to pick Python:

• Rich Ecosystem: Libraries like requests, feedparser, and BeautifulSoup simplify API calls and web scraping.
• Fast Prototyping: Write less boilerplate code compared to compiled languages.
• Scalable Frameworks: Flask and Django provide lightweight and heavyweight options for the backend.
• SEO‑Friendly: Python’s templating engines (Jinja2, Django templates) let you generate clean HTML that search bots love.

Core Features Every Good Aggregator Should Have

Essential Functionality

• Source management (add, edit, remove RSS feeds or API endpoints).
• Automatic fetching and scheduling (e.g., every 30 minutes).
• Article parsing (title, description, author, publish date, image).
• Keyword‑based filtering and categorization.
• User‑friendly UI with pagination or infinite scroll.

Nice‑to‑Have Enhancements

• User authentication and saved preferences.
• Full‑text search powered by Whoosh or Elasticsearch.
• Responsive design using Bootstrap or Tailwind CSS.
• Social sharing buttons and comment integration.
• Analytics dashboard to track most‑read topics.

Tech Stack Overview

Language	Python 3.11+
Web Framework	Flask (lightweight) or Django (full‑featured)
Database	SQLite for prototyping, PostgreSQL for production
Task Scheduler	Celery with Redis broker or APScheduler
Front‑End	HTML5, CSS3, JavaScript (vanilla or Vue/React)
Hosting	Heroku, Render, or AWS Elastic Beanstalk

Step‑by‑Step Development Guide

1. Set Up the Project Structure

news_aggregator/
│
├─ app/
│   ├─ __init__.py
│   ├─ routes.py
│   ├─ models.py
│   └─ utils.py
│
├─ templates/
│   └─ index.html
│
├─ static/
│   ├─ css/
│   └─ js/
│
├─ requirements.txt
└─ run.py

Use a virtual environment and install the core dependencies:

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install flask sqlalchemy requests feedparser APScheduler

2. Define the Database Model

from datetime import datetime
from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()

class Article(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    source = db.Column(db.String(120), nullable=False)
    title = db.Column(db.String(255), nullable=False)
    url = db.Column(db.String(500), nullable=False, unique=True)
    summary = db.Column(db.Text)
    published = db.Column(db.DateTime, default=datetime.utcnow)
    image_url = db.Column(db.String(500))

This simple schema stores the most important metadata for each article while keeping the table lightweight for fast queries.

3. Fetch News from External Sources

Below is a reusable function that pulls data from an RSS feed using feedparser and stores new articles in the database.

import feedparser
from .models import Article, db

def fetch_rss(feed_url):
    feed = feedparser.parse(feed_url)
    for entry in feed.entries:
        # Avoid duplicates
        if Article.query.filter_by(url=entry.link).first():
            continue

        article = Article(
            source=feed.feed.title,
            title=entry.title,
            url=entry.link,
            summary=getattr(entry, 'summary', ''),
            published=getattr(entry, 'published_parsed', None),
            image_url=entry.media_content[0]['url'] if 'media_content' in entry else ''
        )
        db.session.add(article)
    db.session.commit()

For JSON‑based APIs (e.g., NewsAPI.org), replace feedparser with requests.get() and parse the JSON response accordingly.

4. Schedule Automatic Updates

Using APScheduler, you can run fetch_rss every 30 minutes without leaving the server process.

from apscheduler.schedulers.background import BackgroundScheduler
from .utils import fetch_rss

def start_scheduler(app):
    scheduler = BackgroundScheduler()
    scheduler.add_job(
        func=lambda: fetch_rss('https://rss.cnn.com/rss/edition.rss'),
        trigger='interval',
        minutes=30,
        id='cnn_feed'
    )
    scheduler.start()
    # Shut down gracefully when the app exits
    app.teardown_appcontext(lambda exception: scheduler.shutdown())

5. Build the Flask Routes

from flask import Flask, render_template, request
from .models import db, Article
from .utils import start_scheduler

def create_app():
    app = Flask(__name__)
    app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///news.db'
    db.init_app(app)

    @app.route('/')
    def index():
        page = request.args.get('page', 1, type=int)
        articles = Article.query.order_by(Article.published.desc()).paginate(page, 10, False)
        return render_template('index.html', articles=articles)

    with app.app_context():
        db.create_all()
        start_scheduler(app)

    return app

6. Design a Clean Front‑End

Save the following as templates/index.html. It uses Bootstrap CDN for quick styling and includes SEO‑friendly meta tags.

<!DOCTYPE html>

<html lang="en">

<head>

    <meta charset="UTF-8">

    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <meta name="description" content="Live Python‑powered news aggregator pulling top stories from RSS feeds and APIs.">

    <title>Python News Aggregator</title>

    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" rel="stylesheet">

</head>

<body class="bg-light">

    <div class="container py-4">

        <h2 class="mb-4">Latest Headlines</h2>

        <div class="row">

            {% for article in articles.items %}

            <div class="col-md-6 mb-3">

                <div class="card h-100">

                    {% if article.image_url %}

                    <img src="{{ article.image_url }}" class="card-img-top" alt="Article image">

                    {% endif %}

                    <div class="card-body">

                        <h5 class="card-title">

                            <a href="{{ article.url }}" target="_blank" rel="noopener">{{ article.title }}</a>

                        </h5>

                        <p class="card-text">{{ article.summary|truncate(150) }}</p>

                    </div>

                    &