## Directory Structure

Below is the directory and file layout for the `scipaperloader` project:

```plaintext
scipaperloader/
├── app/
│   ├── __init__.py        # Initialize Flask app and database
│   ├── models.py          # SQLAlchemy database models
│   ├── main.py            # Flask routes (main blueprint)
│   ├── templates/         # Jinja2 templates for HTML pages
│   │   ├── base.html      # Base layout template with Alpine.js and HTMX
│   │   ├── index.html     # Home page template
│   │   ├── upload.html    # CSV upload page template
│   │   ├── schedule.html  # Schedule configuration page template
│   │   └── logs.html      # Logs display page template
│   └── static/            # Static files (CSS, JS, images)
├── scraper.py             # Background scraper daemon script
├── tests/
│   └── test_scipaperloader.py  # Tests with a Flask test fixture
├── config.py              # Configuration settings for different environments
├── pyproject.toml         # Project metadata and build configuration
├── setup.cfg              # Development tool configurations (linting, testing)
├── Makefile               # Convenient commands for development tasks
└── .venv/                 # Python virtual environment (not in version control)
```

- The **`app/`** package contains the Flask application code. It includes an `__init__.py` to create the app and set up extensions, a `models.py` defining database models with SQLAlchemy, and a `main.py` defining routes in a Flask Blueprint. The `templates/` directory holds HTML templates (with Jinja2 syntax) and `static/` will contain static assets (e.g., custom CSS or JS files, if any).  
- The **`scraper.py`** is a **standalone** Python script acting as a background daemon. It can be run separately to perform background scraping tasks (e.g., periodically fetching new data). This script will use the same database (via SQLAlchemy models or direct database access) to read or write data as needed.  
- The **`tests/`** directory includes a test file that uses pytest to ensure the Flask app and its components work as expected. A Flask fixture creates an application instance for testing (with an in-memory database) and verifies routes and database operations (e.g., uploading CSV adds records).  
- The **configuration and setup files** at the project root help in development and deployment. `config.py` defines configuration classes (for development, testing, production) so the app can be easily configured. `pyproject.toml` and `setup.cfg` provide project metadata and tool configurations (for packaging, linting, etc.), and a `Makefile` is included to simplify common tasks (running the app, tests, etc.).  

## How to use the logger

### GUI Interactions:

```python
ActivityLog.log_gui_interaction(
    action="view_paper_details",
    description="User viewed paper details",
    paper_id=123
)
```

### Configuration Changes:

```python
ActivityLog.log_gui_interaction(
    action="view_paper_details",
    description="User viewed paper details",
    paper_id=123
)
```

### Scraper Commands:

```python
ActivityLog.log_scraper_command(
    action="start_scraper", 
    status="running"
)
```

### Scraper Activities:

```python
ActivityLog.log_scraper_activity(
    action="download_paper",
    paper_id=123,
    status="success",
    description="Paper downloaded successfully",
    file_path="/papers/123.pdf"
)
```

### Error Logging:

```python
# Simple error
ActivityLog.log_error(
    error_message="Failed to connect to API",
    severity=ErrorSeverity.WARNING.value,
    source="api_client"
)

# Logging an exception
try:
    result = some_risky_operation()
except Exception as e:
    ActivityLog.log_error(
        error_message="Operation failed",
        exception=e,
        severity=ErrorSeverity.ERROR.value,
        source="data_processor",
        paper_id=paper_id
    )
```