7c3847fbb0
Replaces the brittle agent-driven /api/bizportal/search queue with a
deterministic Python FastAPI service that drives BizPortal via a real
headless Chromium. The old flow is untouched.
What this adds:
- scripts/bizportal_service.py — FastAPI + Playwright service with
/search, /reserve, /health endpoints. Persistent login session
via storage_state.json. Locator-based selectors (vs eval JS).
- app/bizportal-2/ — new /bizportal-2 page, synchronous fetch
(no polling), shows round-trip duration, mirrors the existing
bulk-add + link UX.
- app/api/bizportal-2/{search,reserve,health}/route.ts — thin
proxies to the Python sidecar.
- scripts/README.md, .env.example, .gitignore, requirements.txt.
- .gitignore: ignore tsconfig.tsbuildinfo (build artefact).
Wiring:
- components/NavBar.tsx: 'BizPortal 2.0' nav entry with TEST badge.
- app/bizportal/page.tsx: 'Try BizPortal 2.0' action link in
header so the new option is discoverable from the existing page.
Run: cd scripts && source .venv/bin/activate && uvicorn
bizportal_service:app --port 7001
162 lines
4.1 KiB
Markdown
162 lines
4.1 KiB
Markdown
# BizPortal 2.0 — Python Playwright service
|
|
|
|
Sidecar that drives `https://www.bizportal.gov.za` via a real headless
|
|
Chromium. The Next.js side talks to it over HTTP — no LLM agent, no
|
|
file queue, no polling. Sits next to the existing `/bizportal` page
|
|
without touching it.
|
|
|
|
## Why this exists
|
|
|
|
The existing `/bizportal` page queues searches to `/tmp/bizportal_queue/`
|
|
and an LLM agent reads the page snapshot, decides what to click, and
|
|
writes the result back. Fast enough when it works, but the agent is
|
|
the brittle bit — it misreads the page, times out, or clicks the
|
|
wrong button. This service replaces that with a deterministic Python
|
|
script.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Browser ─► /bizportal-2 ─► /api/bizportal-2/* ─► this service ─► BizPortal
|
|
│
|
|
└─► PocketBase (via service)
|
|
```
|
|
|
|
The Python service holds a persistent `storage_state.json` so the
|
|
user is logged in once and every subsequent request reuses the
|
|
session.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
cd scripts
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
playwright install chromium
|
|
cp .env.example .env # fill in real credentials
|
|
```
|
|
|
|
## Run
|
|
|
|
```bash
|
|
# Dev
|
|
uvicorn bizportal_service:app --host 127.0.0.1 --port 7001 --reload
|
|
|
|
# Production (Coolify / systemd)
|
|
uvicorn bizportal_service:app --host 127.0.0.1 --port 7001 --workers 1
|
|
```
|
|
|
|
Keep `--workers 1` — the script uses a single shared browser context
|
|
to preserve the logged-in session.
|
|
|
|
## Endpoints
|
|
|
|
### `GET /health`
|
|
|
|
```json
|
|
{
|
|
"ok": true,
|
|
"service": "bizportal-2",
|
|
"playwright_ready": true,
|
|
"logged_in_cached": true,
|
|
"headless": true
|
|
}
|
|
```
|
|
|
|
### `POST /search`
|
|
|
|
```json
|
|
{ "type": "ck", "query": "2022/808158/08" }
|
|
```
|
|
|
|
```json
|
|
{
|
|
"ok": true,
|
|
"query": "2022/808158/08",
|
|
"type": "ck",
|
|
"results": [
|
|
{
|
|
"enterprise_no": "2022/808158/08",
|
|
"enterprise_name": "AMPLEFORTH TRADING (PTY) LTD",
|
|
"enterprise_type": "Private Company",
|
|
"enterprise_status": "Business is in Business",
|
|
"directors": [
|
|
{
|
|
"id_number": "870922XXXX08X",
|
|
"names": "DAVID",
|
|
"surname": "ELS",
|
|
"status": "Active",
|
|
"type": "Director"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"log": ["Navigating to login", "..."]
|
|
}
|
|
```
|
|
|
|
### `POST /reserve`
|
|
|
|
```json
|
|
{ "names": ["AMPLEFORTH PTY LTD", "AMPLEFORTH TRADING", "AMPLEFORTH SA", "AMPLEFORTH HOLDINGS"], "dry_run": false }
|
|
```
|
|
|
|
```json
|
|
{
|
|
"ok": true,
|
|
"balance_before": 412.50,
|
|
"status": "bizportal_done",
|
|
"log": ["..."],
|
|
"message": "Submitted via Bizportal. The R50 fee will be debited later."
|
|
}
|
|
```
|
|
|
|
## Tunable selectors
|
|
|
|
BizPortal uses ASP.NET WebForms (`cntMain_*` IDs). The known-stable
|
|
IDs for login and name reservation are baked in. The **search** field
|
|
IDs are not 100% verified — try a search after starting the service
|
|
once and tune the `DIRECTOR_SEARCH_FIELD_CANDIDATES` /
|
|
`ENTERPRISE_SEARCH_FIELD_CANDIDATES` lists at the top of
|
|
`bizportal_service.py` if you get a "Could not find …" error.
|
|
|
|
The page URL is logged in the error so you can inspect the live page
|
|
in your browser and grab the right field ID.
|
|
|
|
## Resetting the session
|
|
|
|
```bash
|
|
rm scripts/storage_state.json
|
|
# Restart the service — it will log in once and re-cache the session.
|
|
```
|
|
|
|
## Running as a systemd service
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=BizPortal 2.0 Python service
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=www-data
|
|
WorkingDirectory=/var/www/workflow/scripts
|
|
EnvironmentFile=/var/www/workflow/scripts/.env
|
|
ExecStart=/var/www/workflow/scripts/.venv/bin/uvicorn bizportal_service:app --host 127.0.0.1 --port 7001 --workers 1
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
## What this does NOT replace
|
|
|
|
- The existing `/bizportal` page (kept untouched)
|
|
- The `/api/bizportal/search/*` queue routes (kept untouched)
|
|
- The `/api/register/[id]/apply-bizportal` route (kept untouched)
|
|
- The PocketBase collections and bulk-add / link logic
|
|
|
|
If you decide to retire the old flow, see
|
|
`scripts/README.rollback.md` (TODO if you want one).
|