Skip to content

Transfer: export, import and batch operations

rn_forge.django.drf.transfer needs the transfer extra (django-import-export and rn-forge-data, which brings tablib and openpyxl). It is not imported by rn_forge.django.drf, so a project that does not use it installs neither. Register viewsets with CustomMethodRouter so the extra actions are served as /orders:import and /orders:batchCreate (AIP-136), not /orders/import.

from import_export import fields, resources, widgets
from rest_framework.viewsets import ModelViewSet

from rn_forge.django.drf.routers import CustomMethodRouter
from rn_forge.django.drf.transfer import (
    BatchCreateMixin, BatchDeleteMixin, ResourceExportMixin, ResourceImportMixin,
)


class OrderExport(resources.ModelResource):
    store = fields.Field(attribute="store__sap", column_name="Store")

    class Meta:
        model = Order
        fields = ("reference", "store", "quantity")


class OrderImport(resources.ModelResource):
    reference = fields.Field(attribute="reference", column_name="REF")
    store = fields.Field(
        attribute="store", column_name="STORE",
        widget=widgets.ForeignKeyWidget(Store, field="sap"),
    )

    class Meta:
        model = Order
        fields = ("reference", "store")
        import_id_fields = ("reference",)
        skip_unchanged = True
        report_skipped = True

    def before_import_row(self, row, **kwargs):
        user = kwargs["user"]  # the request user: set audit fields, authorize the row


class OrderViewSet(
    ResourceExportMixin, ResourceImportMixin, BatchCreateMixin, BatchDeleteMixin, ModelViewSet
):
    queryset = Order.objects.all()
    serializer_class = OrderSerializer
    export_resource_class = OrderExport
    import_resource_class = OrderImport
    export_column_formats = {"Quantity": "#,##0"}


router = CustomMethodRouter(trailing_slash=False)
router.register("orders", OrderViewSet, basename="orders")

A viewset takes only the mixins it needs (BatchGetMixin, BatchCreateMixin, BatchUpdateMixin, BatchDeleteMixin). The wire shapes are those of the transfer section of the rn-forge-web API conventions.

Request Behaviour
GET /orders with Accept: text/csv or the XLSX media type, or ?format=csv\|xlsx The JSON list's filters, then Resource.export(); not paginated. Above transfer.max_rows it is a 422 problem naming the cap. Content-Disposition carries filename*. Permission is that of list.
POST /orders:import, multipart/form-data file, ?validateOnly=true Upsert through the import resource. 200 {created, updated, skipped, validateOnly}; any row error is a 422 with errors[].pointer = "/rows/<row>/<column>" and nothing persisted. Permission upload.
GET /orders:importTemplate, ?prefill=true The import columns, negotiated like an export; prefill adds the filtered rows.
POST /orders:batchCreate, {"requests": [...]} Each item through the serializer, all or nothing. 200 {"orders": [...]}.
GET /orders:batchGet?ids=1&ids=2 The ids within filter_queryset(get_queryset()), serialized with the view's serializer. 200 {"orders": [...]} in request order. No ids, or more than transfer.max_rows, is a 400; an id outside the filtered queryset is a 404 naming it.
POST /orders:batchUpdate, {"requests": [{"id", "patch", "ifMatch"}]} Each patch is a JSON Merge Patch merged into the row and validated through the serializer, all or nothing, in one transaction. 200 {"orders": [...]} in request order. A failed precondition is a 428, 400 or 412 at /requests/<i>/ifMatch; an invalid merge is a 422 at /requests/<i>/patch/<field>.
POST /orders:batchDelete, {"ids": [...]} All or nothing, 204. An id outside the filtered queryset is a 404 naming it.

In :batchGet, :batchUpdate and :batchDelete, an id the primary key cannot hold, such as abc for an integer key, is a 404 naming it, like any other id outside the queryset.

Row numbers in pointers count from 0. A pointer names the column as it appears in the file when the failing field is an import field. An error import-export raises outside field validation (a ForeignKeyWidget that finds no row, or an exception in before_import_row) points at the whole row.

Per-item authorization

Override the hooks and return a message to refuse an item; the response is one 403 problem with errors[].pointer naming the item, and nothing is persisted.

  • prepare_batch_create_item(item) runs on each raw item before validation.
  • validate_batch_create_item(item) runs on each validated item, after AuditFieldsViewMixin.prepare_create_data.
  • validate_batch_update_item(instance, data) runs on each row and its validated data, after AuditFieldsViewMixin.prepare_update_data.
  • validate_batch_delete_instance(instance) runs on each instance to delete.

BatchUpdateMixin checks ifMatch for a VersionedModelMixin row, and an unversioned model ignores it. Set batch_update_requires_if_match = True to make a missing ifMatch a 428, and etag_codec to change the validator format, as for MergePatchMixin. BatchGetMixin and BatchUpdateMixin answer under batch_resource_name, which they share with BatchCreateMixin.

In an import resource the same check belongs in before_import_row(row, **kwargs), where kwargs["user"] is the request user; raising there is reported as a row error.

Committing the valid rows

Set import_rollback_on_validation_errors = False on the viewset to commit the valid rows when the only failures are field validation errors. The response is 200 with the failures under errors. Any other error still rolls the import back.

Settings

RN_FORGE_DJANGO["DRF"]["TRANSFER"]["MAX_ROWS"] (default 10 000, None for no cap) limits the rows of an export or import and the items of a batch request.

Migrating from ew-loop-api

The application moves when it adopts pykit; nothing here gates pykit.

ew-loop-api Now
_get_download_template / _build_download_template_row An export Resource: Field(attribute="store__sap", column_name="Store"), dehydrate_<field> for computed columns and enum display names, export_column_formats for dates and currency
_get_upload_template / _build_instance / _build_foreignkey_lookups An import Resource: import_id_fields, ForeignKeyWidget(Role, field="code"), before_import_row for normalization and per-row authorization, skip_unchanged with explicit fields
POST .../download with base64 CSV in a JSON body GET with Accept: text/csv or the XLSX media type, or ?format= on a link
POST .../upload with simulation mode POST /orders:import with ?validateOnly=true
Commit valid rows, report the rest import_rollback_on_validation_errors = False
Bulk create (POST of a JSON list) and bulk delete (DELETE with ids), on hyphenated sub-paths POST /orders:batchCreate with {"requests": [...]}; POST /orders:batchDelete with {"ids": [...]}
Row errors joined with "\n" into a 403 403 or 422 problem, one errors[] entry per item
Bespoke report workbooks Build the workbook in a service and return it with content_disposition from rn_forge.web