Import Engine

This document describes how the ADP Car Market Hub plugin imports vehicle listings into WordPress. The plugin supports two import front-ends that share the same downstream pipeline:

• the AutoScout24 API importer (AS24CI\Client → AS24CI\Importer), documented in detail below, and
• the Connections front-end (internal class AS24CI\Universal_Importer; the admin screen is named "Connections", slug as24ci-universal-import), which imports CSV / XML / XLSX / JSON / ZIP files and named connectors (feed, e-mail, carcuro, mobile.de). See The Connections importer below and the Connections admin reference.

This document covers the responsibilities of the importer classes, the per-listing flow, change detection, full-sync deletion and the public hook that fires after a vehicle is imported.

When to use this document

Read this document if you need to:

• Understand what happens during an import run.
• Diagnose why a vehicle was inserted, updated, skipped or removed.
• Build a custom integration that reacts to imports via the as24ci_vehicle_imported action.
• Plan capacity or hosting requirements for large catalogues.

For scheduling and cron details, see Cron Events And Scheduler. For image handling, see Image Importer And Queue.

Overview

The import engine consists of several classes that work together:

• AS24CI\Client — issues authenticated HTTP requests to the AutoScout24 API, manages OAuth tokens (including the as24ci_access_token transient cache) and exposes helpers for preview pages and per-listing equipment.
• AS24CI\Importer — coordinates per-listing import. Decides whether to insert, update or skip and writes the vehicle's post, postmeta and image references. Owns change detection.
• AS24CI\Mapper — maps the raw AutoScout24 payload to the fields stored in the as24_vehicles table via AS24CI\Vehicle_Repository. Maintains backwards-compatible postmeta keys.
• AS24CI\Vehicle_Repository — persists vehicle field data in the dedicated {$wpdb->prefix}as24_vehicles table.
• AS24CI\Image_Importer — downloads images, optionally converts them to WebP, and attaches them to the vehicle post. See Image Importer And Queue.
• AS24CI\Scheduler — wraps Importer::import_all_for_seller() in a locking, retry-safe runner used by WP-Cron, the REST cron endpoint and the manual "Run now" button.

The same Importer instance is reused by the manual Batch-Wizard, the cron runner and the REST cron endpoint.

Requirements or prerequisites

Before running an import you need:

• Valid AutoScout24 API credentials configured in the plugin (as24ci_base_url, as24ci_token_url, as24ci_client_id, as24ci_client_secret, as24ci_token_audience).
• One or more seller IDs in as24ci_seller_ids (comma-separated).
• A WordPress user designated as the default post author (as24ci_default_post_author); imports fall back to the current user when no default is set.
• Working outbound HTTPS access from the WordPress server to the AutoScout24 API and to the image hosts referenced in the API response.
• Sufficient PHP execution time. The scheduler raises the time limit to 300 seconds via set_time_limit(300) for cron and REST runs; some hosts may override this.

Per-listing flow

AS24CI\Importer::upsert_post_from_listing() is the central per-listing routine. The simplified flow is:

1. Validate the listing has an id. Skip otherwise.
2. Skip listings whose API status is not activated or whose live flag is empty.
3. Look up an existing vehicle post by listing ID (find_post_id_by_listing_id()).
4. Run change detection (see below). If nothing has changed, refresh the last-sync heartbeat via AS24CI\Sync_State::touch() and return skipped.
5. Build the post title, excerpt and content. If the AI Lock (_as24ci_ai_locked = 'yes') is active, do not overwrite the post excerpt or content.
6. Insert or update the WordPress post with the configured default post status and author.
7. Write backwards-compatibility postmeta: _as24ci_listing_id, _as24ci_seller_id and _as24ci_last_modified (when the API supplies it). The last-sync heartbeat is recorded separately via AS24CI\Sync_State::touch() (a dedicated as24ci_sync_state table, not postmeta).
8. Ensure the slug ends with the listing ID (ensure_slug_has_listing_id()).
9. Map the listing payload via AS24CI\Mapper::map_listing_to_post() so the typed fields are written to the as24_vehicles table.
10. Fetch equipment via Client::get_listing_equipment() and map it to the post via Mapper::map_equipment_to_post(). Errors are logged but do not fail the listing.
11. Import images (subject to the as24ci_import_images toggle and as24ci_max_images cap). The image flow is described in detail in Image Importer And Queue.
12. Update _as24ci_content_hash and _as24ci_original_description.
13. Fire the as24ci_vehicle_imported action with the post ID, the raw API listing and a boolean $is_update.
14. Return one of the strings inserted, updated, skipped or error (on wp_insert_post / wp_update_post failure).

Change detection

The importer skips listings that have not changed since the previous run, using two cascading checks:

• Primary: the lastModifiedDate field returned by the API. If the local _as24ci_last_modified postmeta matches the remote value, the listing is treated as unchanged. The importer still refreshes the last-sync heartbeat (AS24CI\Sync_State::touch()) for the Importer admin tab. This heartbeat is purely informational; the full-sync deletion logic relies on listing-ID membership, not on the last-sync timestamp.
• Fallback: when lastModifiedDate is missing, the importer computes an MD5 of the JSON-encoded payload and compares it against the stored _as24ci_content_hash.

Image change detection runs independently and is described in the image importer documentation.

AI Lock

When _as24ci_ai_locked is 'yes', the importer keeps the existing post_content and post_excerpt so AI-generated copy survives a re-import. All other fields, taxonomy assignments, equipment and images are still updated. The original API description is also preserved separately in _as24ci_original_description.

Slug stability

ensure_slug_has_listing_id() updates the post slug to end with -<listing_id> whenever the slug changes. This keeps URLs stable across re-imports and avoids accidental redirects when titles change.

Bulk operations

AS24CI\Importer exposes the following high-level methods:

• get_seller_ids() — returns the cleaned list of configured seller IDs.
• get_all_preview_listings_for_seller( $seller_id, $max_pages = 20, $page_size = 50 ) — pages through the API and returns the full set of activated, live listings for a single seller.
• preview_listings_for_seller( $seller_id, $page = 1, $per_page = 50 ) — paginated preview used by the wizard's UI.
• import_selected_listings( $seller_id, $listing_ids, $max_vehicles = 0 ) — imports the given listing IDs. Honours the $max_vehicles budget; skipped vehicles do not consume budget.
• import_all_for_seller( $seller_id, $max_vehicles = 0 ) — fetches the full preview list for a seller and delegates to import_selected_listings(). Returns counts plus listing_ids (the full set seen this run) and api_active (number of active listings the API returned).
• full_sync_after_import( array $remote_listing_ids ) — deletes local as24ci_car posts whose _as24ci_listing_id is not in the given remote set. Skipped if the remote set is empty as a safety guard against accidental wipe.
• upsert_external_listing( array $listing, string $source_id ) — Universal-Importer entry point: tags the vehicle with the _as24ci_source postmeta and routes the AutoScout24-shaped listing through the same upsert_post_from_listing() flow. Returns inserted, updated, skipped or error.
• full_sync_for_source( string $source_id, array $external_ids ) — full sync scoped to a single connector source: **permanently deletes** only that source's vehicles missing from the latest run, never touching AutoScout24 vehicles or another source. Returns { kept, deleted } (a legacy held key may still appear in the signature but holding is no longer used — see Soft-delete removed); a no-op on an empty source or id.
• hard_delete_listing( $listing_id ) — manual destructive delete for a single listing; goes through the same Vehicle_Deleter cleanup as native WordPress delete.

The "All-in-One" version of the plugin imposes no listing cap; get_free_import_limit() and get_free_slots_left() return PHP_INT_MAX.

The Connections importer

The Connections importer (internal class name AS24CI\Universal_Importer; the admin screen is labelled "Connections") brings non-AutoScout24 inventory into the same pipeline. It is format-agnostic and source-agnostic: any source row is first turned into an AutoScout24-shaped $listing array and then fed into the same AS24CI\Importer used for the API import, so VIN re-listing detection, value normalization, image import, slug stability and team assignment all behave identically.

Reader and format dispatch

AS24CI\Import_Reader::read( string $content, string $filename = '' ) returns a uniform { header, rows } structure and dispatches by format to a per-format reader:

Import_Reader::detect_format() resolves the format from the filename extension first, then falls back to content sniffing (ZIP magic bytes — distinguishing an Office .xlsx from a data+images bundle — then a leading < for XML or {/[ for JSON, otherwise CSV). The accepted extensions are csv, txt, tsv, xml, xlsx, json, zip (Import_Reader::ACCEPTED_EXTENSIONS).

• The CSV reader normalizes the encoding to UTF-8 first (see below), auto-detects the delimiter (comma / semicolon / tab) and parses with fgetcsv on a memory stream so quoted fields that contain the delimiter, quotes or line breaks (multi-line descriptions) are read correctly.
• AS24CI\Import_Encoding::to_utf8() converts DACH-region exports (Windows-1252 / ISO-8859, UTF-16/32 with a BOM) to clean UTF-8. The XLSX reader does not use it (its inner XML is always UTF-8).
• The ZIP reader reads the first supported data file inside the archive, extracts bundled images into an as24ci-import/… folder under wp-content/uploads, and rewrites bare image filenames in the rows to the resulting public URLs.

Mapping template

AS24CI\Import_Template turns one raw source row into an AutoScout24-shaped listing. Canonical field names are the AutoScout24 field names (makeKey, price, …) used by AS24CI\Field_Mapping and AS24CI\Mapper. The template:

• maps configured source columns → canonical fields via field_map;
• normalizes numeric fields: INTEGER_FIELDS (drops thousands separators such as ., ', spaces and trailing units) and FLOAT_FIELDS (accepts a decimal comma, dot output);
• applies per-field value_maps (lower-cased source value → canonical value) and coerces recognised boolean fields;
• builds a stable id: dealer stock number → VIN-<normalised VIN> → ROW-<row hash>;
• splits the configured image column on the image delimiter and promotes scheme-less URLs (e.g. www.example.com/a.jpg) to absolute https://;
• forces status = activated and live = true so the downstream importer does not skip the row.

Run flow

AS24CI\Universal_Importer::run( $content, $template, $source_id, $dry_run = true, $sample_limit = 5, $full_sync = false ):

1. Parses $content via Import_Reader::read().
2. For each row, builds a listing via Import_Template::to_listing() and runs deterministic validation (Universal_Importer::validate() — requires an identity, a make/model/version, a plausible VIN when present and a numeric price).
3. On a dry run, returns a sample preview (new vs. update) without writing.
4. Otherwise writes each valid listing via Importer::upsert_external_listing( $listing, $source_id ), which tags the vehicle with the _as24ci_source postmeta and routes through the shared upsert_post_from_listing(). Manual/file imports always import images immediately (never the queue).
5. When $full_sync is true and a $source_id is set, calls Importer::full_sync_for_source() to permanently delete vehicles of THIS source that were absent from the run. Full-sync is opt-in because a manual upload is often only a subset of the inventory; it never touches AutoScout24 vehicles or another source.

Named connectors

Four connectors reuse the same template + reader to import on a schedule, each storing a settings option plus (where applicable) an encrypted secret option via AS24CI\Secrets:

• AS24CI\Import_Feed — pulls a file from an https://, ftp://, ftps:// or sftp:// URL or a server file path (with Dropbox / Google Drive / Google Sheets / OneDrive share-link normalization), reusing the saved as24ci_import_csv_template mapping. Runs on its own cron event as24ci_import_feed_event.
• AS24CI\Import_Mail — connects over IMAP and imports the best matching attachment from the newest unread message. Runs on its own cron event as24ci_import_mail_event.
• AS24CI\Import_Carcuro — pulls the carcuro public inventory API using a fixed built-in field mapping (Import_Carcuro::carcuro_template()), so no manual column mapping is required. carcuro's accident_free is inverted into the canonical hadAccident flag via a value_map. Runs on the central import schedule (AS24CI\Scheduler::CRON_HOOK = as24ci_scheduled_import); it has no cron event of its own and is a no-op unless carcuro is enabled and a token is set.
• AS24CI\Import_Mobile (includes/class-as24ci-import-mobile.php) — the mobile.de connector. Imports the dealer's mobile.de inventory (JSON) through the built-in Import_Mobile::mobile_template(). Like carcuro, it runs on the central import schedule (as24ci_scheduled_import) rather than a per-source event, and its run() is a no-op unless mobile.de is enabled and credentials are present. Behaviour (schedule, full-sync) is governed centrally under "Import & Limits", exactly like AutoScout24.

The connectors expose a common register_hooks() / run() / run_now() / execute() / finish() surface. For their cron events see Cron Events And Scheduler; credentials are entered in the Connections admin tab (internal class AS24CI\Admin_Tab_Universal_Import, slug as24ci-universal-import), not in Settings.

Vehicle deletion

Every permanent delete path flows through AS24CI\Vehicle_Deleter:

• WordPress native "Delete permanently" on the Cars list.
• Importer full sync (full_sync_after_import()).
• Bulk actions (AS24CI\Bulk_Actions).
• Manual Importer::hard_delete_listing().

The deleter runs on before_delete_post / deleted_post and:

• Removes attachments tracked in _as24ci_image_ids.
• Removes the corresponding row in {$wpdb->prefix}as24_vehicles.
• Fires the as24ci_vehicle_deleted action.

Manual gallery attachments (_as24ci_manual_image_ids) and other attachments not tracked by the importer are intentionally preserved.

Alongside the deleter, AS24CI\Vehicle_Redirects hooks before_delete_post at priority 5 (i.e. before the deleter) so the soon-to-be-removed vehicle's slug and as24ci_brand / as24ci_model term slugs are captured while the post and its terms still exist. The deleted URL is then 301-redirected to a live make/model search on the next request instead of returning a 404 — see architecture overview for the full mechanism.

Soft-delete removed

Earlier versions held source-removed vehicles in a dedicated as24ci_archived ("Removed from source") post status for a grace period before deleting them. **This soft-delete feature has been removed.** Full sync (full_sync_after_import() and full_sync_for_source()) deletes source-removed vehicles immediately through AS24CI\Vehicle_Deleter — delete_post_with_attachments() calls Vehicle_Deleter::delete_vehicle_permanently() unconditionally.

The AS24CI\Soft_Delete class, the as24ci_archived post status, the daily cleanup cron as24ci_soft_delete_cleanup_cron and the as24ci_sync_soft_delete / as24ci_sync_grace_days options **no longer exist**.

A one-time data migration purges any remnants from the old model: it permanently deletes any vehicles still parked in the as24ci_archived status or carrying the _as24ci_soft_deleted_at postmeta (routed through AS24CI\Vehicle_Deleter so attachments and the vehicles-table row are cleaned up too) and unschedules the old as24ci_soft_delete_cleanup_cron cron for installations upgraded from older versions.

In short: with full sync enabled, a vehicle that disappears from its source is hard-deleted on the next run, not held.

Public hooks

The import engine fires the following hooks. Verify hook signatures in the current plugin version before relying on them.

The plugin also reads the as24ci_webp_quality filter when converting images; see the image importer documentation.

Configuration reference

Operational notes

• The importer never throws on a single bad listing. Errors from wp_insert_post, wp_update_post, Client::get_listing_equipment and the image importer are logged via AS24CI\Logger and counted but do not abort the run.
• Change detection means a full re-import of an unchanged catalogue is inexpensive: most listings are skipped and image downloads are not re-issued.
• The mapper writes to the as24_vehicles table via the repository, not to wp_postmeta, except for the small backwards-compat key set documented in Data Model.
• Full-sync is opt-in. If as24ci_full_sync is enabled, ensure the configured seller IDs cover the entire catalogue you want to keep online; vehicles whose listings disappear from the API are permanently deleted.
• Verbose logging produces detailed log lines and grows quickly on large catalogues. The plugin enforces a 10 MB log cap with rotation; adjust verbose logging to balance diagnostics with storage.

Troubleshooting

• No vehicles imported. Confirm the API credentials, that as24ci_seller_ids is non-empty, and that the API returns listings with status = activated and live = true. Check the plugin log for HTTP errors from AS24CI\Client.
• Listings remain in draft status. The plugin defaults as24ci_default_post_status to draft so administrators can review mappings. Change the option to publish once you are satisfied, or publish individual vehicles from the admin list.
• Repeated re-imports keep updating the same vehicles. Verify that the API supplies a stable lastModifiedDate. When it is missing, the fallback content hash detects changes; the hash will change whenever the API payload changes for any reason.
• A vehicle disappeared after a cron run. Check whether as24ci_full_sync is enabled. With full sync on, any listing missing from the API is permanently deleted (its imported attachments included).
• Importer ran but no images were downloaded. Check as24ci_import_images and as24ci_max_images. In cron/REST runs with as24ci_cron_image_queue = 1, only the first image is downloaded immediately and the rest are deferred to the image queue worker.
• Skipping listing_id=… : lastModifiedDate unchanged. This is the expected verbose log line for unchanged listings; it confirms change detection is working.

Related documents

• Image Importer And Queue
• Cron Events And Scheduler
• Architecture Overview
• Data Model
• Database Schema
• Section overview