DataMind

03 / Data Flow

Dataset lifecycle, end to end

The canonical path a dataset travels from upload to a downloadable report.

STEP 1

Upload & Ingest

A CSV/JSON file is uploaded, or fetched via direct URL / the Kaggle API through the Dataset Fetch Service. The raw file is persisted and a Dataset row is created.

STEP 2

Profiling

The profiling service computes a schema summary, stats summary, correlation summary and sample rows — metadata only, never raw rows beyond a bounded sample — stored as a DatasetProfile.

STEP 3

Research Question Generation

question_generator calls the LLM once per dataset to produce a batch of dataset-specific questions, split into Pre-processing and EDA categories, each with target columns, expected output type and rationale. RAG retrieval can ground this call with insights from similarly-shaped past datasets.

STEP 4

Per-Question Analysis Run

For each selected ResearchQuestion, an AnalysisRun starts the LangGraph pipeline: code_generator writes pandas code from the question + profile context.

STEP 5

Sandboxed Execution & Self-Correction

sandbox_execute runs the code as an isolated subprocess (fresh working dir, CPU/memory rlimits, wall-clock timeout, network disabled best-effort). On failure, code_corrector rewrites the code using the traceback and loops back to execution — until success or max_attempts is exhausted, at which point the run is marked failed.

STEP 6

Result Analysis

result_analyzer is a deterministic (non-LLM) node that truncates/sanitizes stdout (max 4000 chars) so downstream LLM calls get clean, bounded input — logic kept in one place instead of duplicated across nodes.

STEP 7

Insight Writing & Visualization

insight_writer turns the execution output into a narrative Insight. visualization_builder asks the LLM only for chart metadata (type/title/labels) — the PNG itself is rendered by the sandboxed script per the code-generator's contract (./output/chart.png) — and is skipped entirely if no chart was produced.

STEP 8

RAG Indexing (best-effort)

On a successful run, the new insight is written into the local Chroma vector store by the RAG indexer, keyed on the dataset's "schema text" embedding — so future datasets with a similar shape can retrieve it. Indexing failure never fails the run itself.

STEP 9

Report & Export

Once enough runs complete, a Report is generated summarizing insights and visualizations for the dataset, downloadable alongside the cleaned/pre-processed dataset.

04 / Supported Modules & Agent Nodes

7 LangGraph nodes, one job each

Each node is a focused async function operating on a shared AnalysisState TypedDict — LLM-calling nodes load their system prompt from a dedicated markdown file under app/agents/prompts/.

question_generator

Runs once per dataset (not per run), invoked directly from POST /datasets/{id}/questions. Produces a structured ResearchQuestionBatch split into Pre-processing and EDA categories, each tagged with target columns and rationale.

code_generator

First node of the per-run graph. Writes pandas analysis code for one research question, using dataset profile context (schema/stats/correlation summaries + sample rows) and, optionally, retrieved similar past insights.

sandbox_execute

Runs the latest generated code in the local, Docker-free sandbox (services/sandbox.py) against the real dataset file, capturing stdout/stderr, exit code, duration and any produced output files (e.g. chart.png).

route_after_execution (conditional edge)

Branches on the execution result: success → result_analyzer; failure with attempts remaining → code_corrector; attempts exhausted → give_up, which marks the run failed and stores the last stderr.

code_corrector

Given the failing code and its traceback, asks the LLM for a corrected version, appends it to code_history, and loops back to sandbox_execute — the self-correction retry loop, bounded by max_attempts.

result_analyzer

Deterministic, no LLM call. Truncates/sanitizes the successful execution's stdout to a bounded size so both insight_writer and visualization_builder receive clean input without duplicating that logic.

insight_writer

Converts the sanitized execution output into a natural-language Insight row tied to the research question and run.

visualization_builder

Asks the LLM only for structured chart metadata (type, title, axis labels) describing a chart the sandboxed script already rendered to disk; the node is skipped entirely when no chart file was produced.

RAG store (ChromaDB, local)

A Docker-free, local persistent Chroma client paired with a local sentence-transformers embedding model — indexing and retrieval cost zero API tokens, which matters because RAG runs on every question-generation and insight-writing call.

RAG retriever

Finds past insights whose source dataset had a similar schema shape, using a synthetic "schema text" (column names + dtypes) embedding as an approximation of dataset similarity — an intentionally simple heuristic, documented for future replacement.

RAG indexer

Writes each completed insight into the local store right after its AnalysisRun succeeds and commits. Best-effort by design — a failure here must never fail the run itself.

Dataset Fetch Service

Handles ingestion from a direct URL or the Kaggle API, normalizing both into the same on-disk raw-dataset representation the rest of the pipeline expects.

LLM Provider Factory

A single get_llm(model_name) factory; active provider is Ollama (fully local, zero API keys, models like gemma3:4b/phi4-mini/qwen2.5:3b), with commented-out swap points for Groq, OpenAI and Gemini for production/evaluation runs.

🔐 Auth

Lightweight local-user login (LocalUser model), JWT-style session for a single-user research prototype.

📡 Live Status

A WebSocket endpoint (api/ws.py) pushes job/run status updates to the frontend in real time as the graph progresses.

📊 Run Metrics

A dedicated RunMetrics table captures token cost and outcome per run — feeding the RAG-vs-baseline research comparison.

🧹 Cleaned Export

Downloadable cleaned/pre-processed dataset and an auto-generated summary report, per the project's stated feature set.

Method	Endpoint	Purpose
POST	/auth/login	Authenticate local user, issue session
POST	/datasets	Upload/register a new dataset
GET	/datasets	List datasets
GET	/datasets/{dataset_id}	Fetch a single dataset
GET	/datasets/{dataset_id}/profile	Schema/stats/correlation profile
POST	/datasets/{dataset_id}/questions	Trigger LLM research-question generation
GET	/datasets/{dataset_id}/questions	List generated research questions
POST	/datasets/{dataset_id}/runs	Kick off one or more per-question analysis runs (LangGraph)
GET	/datasets/{dataset_id}/runs	List analysis runs and their status
GET	/datasets/{dataset_id}/insights	List insights produced for the dataset
GET	/datasets/{dataset_id}/insights/{insight_id}/visualization	Fetch the chart metadata tied to an insight
POST	/datasets/{dataset_id}/report	Generate the dataset summary report
GET	/datasets/{dataset_id}/report	Fetch the generated report
WS	/ws	Live push of job/run status to the frontend

06 / Tech Stack & Tooling

Local-first by design

Every layer was deliberately chosen to run Docker-free and token-cost-free where possible, given the project's API-budget constraints as a research prototype.

Frontend

Next.js 16.2.12App Router

React 19.2.4+ React DOM

TypeScript 5strict typing

Tailwind CSS v4@tailwindcss/postcss

next-themesdark/light theming

lucide-reacticon set

Backend & Orchestration

FastAPI 0.141+ Uvicorn

LangGraph 1.2.10agent state graph

LangChain 1.3.14+ langchain-ollama/groq/openai/google-genai

SQLAlchemy 2 + Alembicasync ORM + migrations

SQLite (aiosqlite)single-file DB — Postgres dropped

Pydantic v2schemas + settings

AI, RAG & Data

Ollamaactive local LLM provider

Groq / OpenAI / Geminiswappable cloud providers

ChromaDB 0.5.23local persistent vector store

sentence-transformerslocal embeddings

pandas / numpysandboxed analysis code

Kaggle APIdataset ingestion

📝 Note on stack evolution

Earlier planning targeted PostgreSQL + pgvector and a paid LLM API (per the project's own memory notes). The shipped local-pipeline milestone consolidated onto SQLite + Ollama + ChromaDB — a fully local, zero-API-cost stack better suited to iterating on the RAG-vs-baseline research comparison without burning a token budget, with the cloud providers kept as drop-in swaps for the final evaluation run.

08 / Contributors & Commit History

Two-person research build

Derived from the full git commit tree (git log --author, --numstat, --name-only) across 19 commits on main, spanning Jul 17 – Sep 11, 2026.

Swayam Patel

Backend & agent pipeline lead

9commits

6.9k+lines

1.1k−lines

Owned the entire backend: project bootstrap, phased backend build-out, and the full LangGraph multi-agent pipeline + RAG integration.

Initial commit & README (as "SmartBI", later renamed to DataMind)
Backend foundation — phase 1
Backend phase 3 (partial) — handed over for API integration
Full multi-agent LangGraph pipeline, RAG integration, dataset workflows
Local pipeline setup: Ollama + ChromaDB + SQLite, plus setup/testing guides

Jalisa Malik

Frontend & API integration lead

10commits

13.1k+lines

0.3k−lines

Built the entire frontend from scratch through to live-data integration — auth, every core screen, and the final backend-polling wire-up.

Frontend setup + login page, auth flow
Home screen (post-login), dataset upload screen
Dataset profile view, RQ selection panel
Job status view + insights dashboard, export panel + report summary view
UI/theme/button polish; final dataset API integration with backend polling

📌 Reading the history

The project began July 17, 2026 as "SmartBI" (renamed DataMind the same day). Jalisa built out the complete frontend against mocked data through late July/early August, while Swayam's backend work proceeded in phases (foundation → partial phase-3 handover) before a large Sep 11 commit landed the full LangGraph pipeline, RAG integration, and a local-only Ollama/ChromaDB/SQLite setup — bringing backend and frontend into alignment for the first time.

09 / Interactive Preview

How it works, in real time

An illustrative walkthrough of the core screens and agent behavior, built from the actual state machine and prompts above (sample data — not a live connection to the running app).

datamind.local

18,204Rows

14Columns

3.2%Missing values

2High-correlation pairs

Column	Dtype	Nulls	Notes
order_date	datetime	0%	Range: 2023-01 → 2026-06
unit_price	float64	0.4%	Right-skewed, 3 outliers
customer_segment	category	1.1%	5 unique values
discount_pct	float64	6.7%	Correlates with churn_flag (0.61)

Generated research questions

Pre-processing

Should rows with missing discount_pct be imputed or dropped? Are the 3 unit_price outliers data-entry errors or legitimate bulk orders?

EDA

Does discount_pct predict churn_flag across customer segments? How has average order value trended month-over-month since 2023?

Run trace — "Does discount_pct predict churn_flag?"

attempt 1 → code_generator → sandbox_execute
  ✗ KeyError: 'churn_flag' not found (column is 'churned')

attempt 2 → code_corrector (reads traceback) → sandbox_execute
  ✓ exit_code 0, duration 842ms, output_files: ['output/chart.png']

→ result_analyzer → insight_writer → visualization_builder → END

✓ Self-correction loop recovered from a column-name mismatch in a single retry — no human intervention.

Insight

Customers with a discount above 20% churn at 2.3× the rate of those below it — strongest in the "SMB" segment, weakest in "Enterprise". Correlation: 0.61.

Chart	Type	Status
discount_vs_churn.png	Grouped bar	Rendered
order_value_trend.png	Line	Rendered

RAG-grounded question generation

Retrieved past insight	Source dataset shape	Similarity
"Discount depth correlates with early churn in subscription data"	12 cols, similar dtypes	0.84
"Outlier prices cluster around bulk-order thresholds"	9 cols, partial overlap	0.62

ℹ Retrieval is schema-shape based (column names + dtypes embedded via a local sentence-transformer) — this is the RAG condition being evaluated against a no-RAG baseline for the research paper's relevance/actionability metrics.

What DataMind does

📁Upload & Profile

🧠Autonomous Questioning

🔁Self-Correcting Analysis

FastAPI backend, Next.js frontend, local-first AI stack

🧭 The one architectural rule the codebase enforces