feat: Implement cron job queue system based on database
- Added PHP support to project configuration. - Updated FTP configuration to exclude additional directories. - Changed remote database host in config.php and enabled debug mode. - Removed outdated TODO from documentation and created a new CRON_QUEUE_PLAN.md. - Introduced a new cron job queue system using database tables pp_cron_jobs and pp_cron_schedules. - Refactored cron job orchestration to improve management and reliability. - Updated OrderAdminService to use the new queue system and removed old file-based logic. - Added migration scripts for new database structure.
This commit is contained in:
@@ -48,7 +48,26 @@
|
||||
"Bash(python3:*)",
|
||||
"Bash(python:*)",
|
||||
"Bash(grep:*)",
|
||||
"Bash(grep ^<b>ver:*)"
|
||||
"Bash(grep ^<b>ver:*)",
|
||||
"Bash(claude mcp:*)",
|
||||
"mcp__serena__get_current_config",
|
||||
"Bash(cd \"C:\\\\visual studio code\\\\projekty\\\\shopPRO\" && npx sass admin/layout/style-scss/style.scss admin/layout/style-css/style.css --no-source-map 2>&1 || sass admin/layout/style-scss/style.scss admin/layout/style-css/style.css --no-source-map 2>&1)",
|
||||
"Bash(echo no 7z:*)",
|
||||
"Bash(cd \"C:/visual studio code/projekty/shopPRO\" && php -r \"\n\\\\$files = [\n 'admin/templates/integrations/logs.php',\n 'admin/templates/site/main-layout.php',\n 'autoload/Domain/Integrations/IntegrationsRepository.php',\n 'autoload/admin/Controllers/IntegrationsController.php',\n];\n\\\\$zipPath = 'updates/0.30/ver_0.310.zip';\nif \\(!is_dir\\('updates/0.30'\\)\\) mkdir\\('updates/0.30', 0777, true\\);\nif \\(file_exists\\(\\\\$zipPath\\)\\) unlink\\(\\\\$zipPath\\);\n\\\\$zip = new ZipArchive\\(\\);\nif \\(\\\\$zip->open\\(\\\\$zipPath, ZipArchive::CREATE\\) !== true\\) { echo 'Cannot create ZIP'; exit\\(1\\); }\nforeach \\(\\\\$files as \\\\$f\\) {\n if \\(file_exists\\(\\\\$f\\)\\) {\n \\\\$zip->addFile\\(\\\\$f, \\\\$f\\);\n echo \\\\\"Added: \\\\$f\\\\n\\\\\";\n } else {\n echo \\\\\"MISSING: \\\\$f\\\\n\\\\\";\n }\n}\n\\\\$zip->close\\(\\);\necho \\\\\"ZIP created: \\\\$zipPath \\(\\\\\".filesize\\(\\\\$zipPath\\).\\\\\" bytes\\)\\\\n\\\\\";\n\" 2>&1)",
|
||||
"Bash(where jar:*)",
|
||||
"Bash(echo php not found:*)",
|
||||
"Bash(/c/xampp/php/php:*)",
|
||||
"Bash(curl:*)",
|
||||
"Bash(cat:*)",
|
||||
"Bash(rm -rf \"/c/visual studio code/projekty/shopPRO/temp/temp_313\" && cd \"/c/visual studio code/projekty/shopPRO\" && powershell -File ./build-update.ps1 -FromTag v0.312 -ToTag v0.313 -ChangelogEntry \"FIX - sync płatności Apilo \\(int cast na apilo_order_id PPxxxxxx dawał 0\\) + logowanie decyzji sync do pp_log\" 2>&1)",
|
||||
"Bash(which php:*)",
|
||||
"mcp__serena__replace_symbol_body",
|
||||
"mcp__serena__insert_after_symbol",
|
||||
"Bash(php:*)",
|
||||
"Bash(rm -rf \"C:/visual studio code/projekty/shopPRO/temp/temp_314\" && cd \"C:/visual studio code/projekty/shopPRO\" && powershell -ExecutionPolicy Bypass -File build-update.ps1 -FromTag v0.313 -ToTag v0.314 -ChangelogEntry \"FIX - naprawa globalnej wyszukiwarki admin \\(Content-Type, Cache-Control, POST, try/catch\\), NEW - title strony z numerem zamówienia\" 2>&1)",
|
||||
"mcp__serena__initial_instructions",
|
||||
"mcp__serena__list_memories",
|
||||
"mcp__serena__find_referencing_symbols"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -27,6 +27,7 @@ project_name: "shopPRO"
|
||||
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
|
||||
languages:
|
||||
- typescript
|
||||
- php
|
||||
|
||||
# the encoding used by text files in the project
|
||||
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
|
||||
|
||||
6
.vscode/ftp-kr.json
vendored
6
.vscode/ftp-kr.json
vendored
@@ -14,6 +14,10 @@
|
||||
".git",
|
||||
".svn",
|
||||
"/.vscode",
|
||||
"/temp/*"
|
||||
"/temp",
|
||||
"/.serena",
|
||||
"/.claude",
|
||||
"/docs",
|
||||
"/tests"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
<?php
|
||||
$database['host'] = 'localhost';
|
||||
$database['remote_host'] = 'host700513.hostido.net.pl';
|
||||
$database['remote_host'] = 'host117523.hostido.net.pl';
|
||||
$database['user'] = 'host117523_shoppro';
|
||||
$database['password'] = 'mhA9WCEXEnRfTtbN33hL';
|
||||
$database['name'] = 'host117523_shoppro';
|
||||
@@ -13,7 +13,7 @@ $config['redis']['host'] = '127.0.0.1';
|
||||
$config['redis']['port'] = 20470;
|
||||
$config['redis']['password'] = 'Gi7FzWtkry19hZ1BqT1LKEWfwokQpigh';
|
||||
|
||||
$config['debug']['apilo'] = false;
|
||||
$config['debug']['apilo'] = true;
|
||||
|
||||
$config['trustmate']['enabled'] = true;
|
||||
$config['trustmate']['uid'] = '34eb36ba-c715-4cdc-8707-22376c9f14c7';
|
||||
|
||||
156
docs/CRON_QUEUE_PLAN.md
Normal file
156
docs/CRON_QUEUE_PLAN.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Plan: System kolejki zadań cron oparty o bazę danych
|
||||
|
||||
## Kontekst
|
||||
|
||||
Obecny system cron ma dwa problemy:
|
||||
1. **Kolejka plikowa (JSON)** — sync płatności/statusów Apilo trzymany w `/temp/apilo-sync-queue.json` — kruchy, brak transakcji, ryzyko utraty danych
|
||||
2. **Monolityczny cron.php** (~550 linii) — brak priorytetów, brak retry z backoff, brak centralnego zarządzania
|
||||
|
||||
Cel: Zastąpienie całego systemu cron tabelą `pp_cron_jobs` z priorytetami, retry/backoff i harmonogramem `pp_cron_schedules`.
|
||||
|
||||
## Nowe pliki
|
||||
|
||||
| Plik | Opis |
|
||||
|------|------|
|
||||
| `autoload/Domain/CronJob/CronJobType.php` | Stałe typów zadań i priorytetów |
|
||||
| `autoload/Domain/CronJob/CronJobRepository.php` | CRUD na `pp_cron_jobs` + `pp_cron_schedules` |
|
||||
| `autoload/Domain/CronJob/CronJobProcessor.php` | Orkiestracja: pobierz zadanie → wywołaj handler → obsłuż wynik |
|
||||
| `tests/Unit/Domain/CronJob/CronJobTypeTest.php` | Testy stałych |
|
||||
| `tests/Unit/Domain/CronJob/CronJobRepositoryTest.php` | Testy repozytorium |
|
||||
| `tests/Unit/Domain/CronJob/CronJobProcessorTest.php` | Testy procesora |
|
||||
| `migrations/0.315.sql` | CREATE TABLE + INSERT harmonogramów |
|
||||
|
||||
## Modyfikowane pliki
|
||||
|
||||
| Plik | Zmiana |
|
||||
|------|--------|
|
||||
| `cron.php` | Zastąpienie ~550 linii orchestratorem (~100 linii) z rejestracją handlerów |
|
||||
| `cron/cron-xml.php` | Usunięcie — logika przeniesiona do handlera `google_xml_feed` |
|
||||
| `cron-turstmate.php` | Usunięcie — logika przeniesiona do handlera `trustmate_invitation` |
|
||||
| `autoload/Domain/Order/OrderAdminService.php` | `queueApiloSync()` → enqueue do DB; usunięcie metod plikowych; `syncApiloPayment()`/`syncApiloStatus()` → public |
|
||||
| `tests/Unit/Domain/Order/OrderAdminServiceTest.php` | Refaktor testów kolejki: mock `CronJobRepository` zamiast pliku JSON |
|
||||
| `docs/DATABASE_STRUCTURE.md` | Dodanie tabel `pp_cron_jobs`, `pp_cron_schedules` |
|
||||
| `docs/CHANGELOG.md` | Wpis o nowym systemie |
|
||||
|
||||
## Schemat DB (`migrations/0.315.sql`)
|
||||
|
||||
### `pp_cron_jobs`
|
||||
```sql
|
||||
CREATE TABLE pp_cron_jobs (
|
||||
id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
|
||||
job_type VARCHAR(50) NOT NULL,
|
||||
status ENUM('pending','processing','completed','failed','cancelled') NOT NULL DEFAULT 'pending',
|
||||
priority TINYINT UNSIGNED NOT NULL DEFAULT 100, -- niższy = ważniejszy
|
||||
payload TEXT NULL, -- JSON z danymi zadania
|
||||
result TEXT NULL, -- JSON z wynikiem
|
||||
attempts SMALLINT UNSIGNED NOT NULL DEFAULT 0,
|
||||
max_attempts SMALLINT UNSIGNED NOT NULL DEFAULT 10,
|
||||
last_error VARCHAR(500) NULL,
|
||||
scheduled_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
started_at DATETIME NULL,
|
||||
completed_at DATETIME NULL,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
INDEX idx_status_priority_scheduled (status, priority, scheduled_at),
|
||||
INDEX idx_job_type (job_type),
|
||||
INDEX idx_status (status)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
|
||||
```
|
||||
|
||||
### `pp_cron_schedules`
|
||||
```sql
|
||||
CREATE TABLE pp_cron_schedules (
|
||||
id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
|
||||
job_type VARCHAR(50) NOT NULL UNIQUE,
|
||||
interval_seconds INT UNSIGNED NOT NULL,
|
||||
priority TINYINT UNSIGNED NOT NULL DEFAULT 100,
|
||||
max_attempts SMALLINT UNSIGNED NOT NULL DEFAULT 3,
|
||||
payload TEXT NULL,
|
||||
enabled TINYINT(1) NOT NULL DEFAULT 1,
|
||||
last_run_at DATETIME NULL,
|
||||
next_run_at DATETIME NULL,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
INDEX idx_enabled_next_run (enabled, next_run_at)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
|
||||
```
|
||||
|
||||
## Typy zadań i priorytety
|
||||
|
||||
| Typ | Priorytet | Harmonogram |
|
||||
|-----|-----------|-------------|
|
||||
| `apilo_token_keepalive` | 10 (krytyczny) | co 4 min |
|
||||
| `apilo_send_order` | 50 (wysoki) | co 1 min |
|
||||
| `apilo_sync_payment` | 50 (wysoki) | event-driven (enqueue przy zmianie) |
|
||||
| `apilo_sync_status` | 50 (wysoki) | event-driven |
|
||||
| `apilo_product_sync` | 100 (normalny) | co 10 min |
|
||||
| `apilo_pricelist_sync` | 100 (normalny) | co 1h |
|
||||
| `apilo_status_poll` | 100 (normalny) | co 10 min |
|
||||
| `price_history` | 100 (normalny) | co 24h |
|
||||
| `order_analysis` | 100 (normalny) | co 10 min |
|
||||
| `trustmate_invitation` | 200 (niski) | co 10 min |
|
||||
| `google_xml_feed` | 200 (niski) | co 1h |
|
||||
|
||||
## Architektura klas
|
||||
|
||||
### CronJobRepository — metody kluczowe
|
||||
- `enqueue($jobType, $payload, $priority, $maxAttempts, $scheduledAt)` — dodaj do kolejki
|
||||
- `fetchNext($limit)` — atomowe pobranie pending jobs (UPDATE WHERE status='pending')
|
||||
- `markCompleted($jobId, $result)` / `markFailed($jobId, $error, $backoffSeconds)`
|
||||
- `hasPendingJob($jobType, $payloadMatch)` — zapobiega duplikatom
|
||||
- `cleanup($olderThanDays)` — GC starych wpisów
|
||||
- `recoverStuck($olderThanMinutes)` — reset stuck "processing" jobs
|
||||
- `getDueSchedules()` / `touchSchedule($id)` — harmonogram
|
||||
|
||||
### CronJobProcessor — orkiestracja
|
||||
- `registerHandler($jobType, callable)` — rejestracja handlera
|
||||
- `createScheduledJobs()` — tworzy jobs z harmonogramów których `next_run_at <= NOW`
|
||||
- `processQueue($limit)` — pobierz + wywołaj handler + markCompleted/markFailed
|
||||
- `run($limit)` — główna metoda: schedules + process
|
||||
|
||||
### Exponential backoff
|
||||
```
|
||||
Próba 1: 60s, Próba 2: 120s, Próba 3: 240s, ... max 3600s (1h)
|
||||
```
|
||||
|
||||
### Zależność "order not yet in Apilo"
|
||||
Handler `apilo_sync_payment`/`apilo_sync_status` sprawdza `apilo_order_id`. Jeśli brak → zwraca false → `markFailed()` z backoffem → zadanie wraca do kolejki. Max 50 prób.
|
||||
|
||||
## Nowy cron.php (schemat)
|
||||
|
||||
```php
|
||||
$cronRepo = new \Domain\CronJob\CronJobRepository($mdb);
|
||||
$processor = new \Domain\CronJob\CronJobProcessor($mdb, $cronRepo);
|
||||
|
||||
// Rejestracja handlerów (każdy to callable)
|
||||
$processor->registerHandler('apilo_token_keepalive', function($payload) use ($integrationsRepo) { ... });
|
||||
$processor->registerHandler('apilo_send_order', function($payload) use ($orderService, ...) { ... });
|
||||
// ... inne handlery
|
||||
|
||||
$result = $processor->run(20);
|
||||
```
|
||||
|
||||
## Zmiany w OrderAdminService
|
||||
|
||||
1. `queueApiloSync()` → `CronJobRepository::enqueue()` zamiast zapisu do pliku JSON
|
||||
2. Usunięcie: `loadApiloSyncQueue()`, `saveApiloSyncQueue()`, `apiloSyncQueuePath()`, stała `APILO_SYNC_QUEUE_FILE`
|
||||
3. `syncApiloPayment()`, `syncApiloStatus()` → zmiana z `private` na `public`
|
||||
4. Jednorazowa migracja: odczyt JSON → insert do DB → usunięcie pliku
|
||||
|
||||
## Kolejność implementacji
|
||||
|
||||
1. Migracja SQL
|
||||
2. `CronJobType.php`
|
||||
3. `CronJobRepository.php` + testy
|
||||
4. `CronJobProcessor.php` + testy
|
||||
5. Modyfikacja `OrderAdminService` (queue → DB, public methods)
|
||||
6. Jednorazowa migracja pliku JSON → DB
|
||||
7. Nowy `cron.php` z handlerami (ekstrakcja logiki z bloków proceduralnych)
|
||||
8. Aktualizacja testów OrderAdminService
|
||||
9. Dokumentacja (DATABASE_STRUCTURE.md, CHANGELOG.md)
|
||||
|
||||
## Weryfikacja
|
||||
|
||||
1. Uruchomienie pełnego zestawu testów: `./test.ps1`
|
||||
2. Sprawdzenie czy nowe testy CronJob* przechodzą
|
||||
3. Sprawdzenie czy istniejące testy OrderAdminService przechodzą po refaktorze
|
||||
4. Weryfikacja migracji SQL na pustej bazie
|
||||
@@ -1 +0,0 @@
|
||||
1. Może warto przepisać zadania cron, na tabelę, tak żeby zadania cron się kolejkowały i wykonywały za podstawie wpisów z bazy danych, takż żeby własnie nie było sytuacji, że zamówienie będzie próbowało zmienić status w apilo, zanim to zamówienie tam trafi. Czy wszystkie takie zadania trafiałyby do kolejki i wykonywałyby się chronologicznie. Ewentualnie jakieś inne podejście możesz zaproponować.
|
||||
Reference in New Issue
Block a user