SciPaperLoader/DEVELOPMENT.md

Directory Structure

Below is the directory and file layout for the scipaperloader project:

scipaperloader/
├── app/
│   ├── __init__.py        # Initialize Flask app and database
│   ├── models.py          # SQLAlchemy database models
│   ├── main.py            # Flask routes (main blueprint)
│   ├── templates/         # Jinja2 templates for HTML pages
│   │   ├── base.html      # Base layout template with Alpine.js and HTMX
│   │   ├── index.html     # Home page template
│   │   ├── upload.html    # CSV upload page template
│   │   ├── schedule.html  # Schedule configuration page template
│   │   └── logs.html      # Logs display page template
│   └── static/            # Static files (CSS, JS, images)
├── scraper.py             # Background scraper daemon script
├── tests/
│   └── test_scipaperloader.py  # Tests with a Flask test fixture
├── config.py              # Configuration settings for different environments
├── pyproject.toml         # Project metadata and build configuration
├── setup.cfg              # Development tool configurations (linting, testing)
├── Makefile               # Convenient commands for development tasks
└── .venv/                 # Python virtual environment (not in version control)

• The app/ package contains the Flask application code. It includes an __init__.py to create the app and set up extensions, a models.py defining database models with SQLAlchemy, and a main.py defining routes in a Flask Blueprint. The templates/ directory holds HTML templates (with Jinja2 syntax) and static/ will contain static assets (e.g., custom CSS or JS files, if any).
• The scraper.py is a standalone Python script acting as a background daemon. It can be run separately to perform background scraping tasks (e.g., periodically fetching new data). This script will use the same database (via SQLAlchemy models or direct database access) to read or write data as needed.
• The tests/ directory includes a test file that uses pytest to ensure the Flask app and its components work as expected. A Flask fixture creates an application instance for testing (with an in-memory database) and verifies routes and database operations (e.g., uploading CSV adds records).
• The configuration and setup files at the project root help in development and deployment. config.py defines configuration classes (for development, testing, production) so the app can be easily configured. pyproject.toml and setup.cfg provide project metadata and tool configurations (for packaging, linting, etc.), and a Makefile is included to simplify common tasks (running the app, tests, etc.).