Big datasets export#18
Open
yaroslav8765 wants to merge 5 commits into
Open
Conversation
AdminForth/1796/image
… aws s3 multipart upload and background jobs plugin AdminForth/1796/image
… remove debug logs AdminForth/1796/image
…f the DECIMAL datatype AdminForth/1796/image
AdminForth/1796/image
There was a problem hiding this comment.
Pull request overview
This PR adds a “big dataset” CSV export mode to the import-export plugin, shifting large exports into a background job that streams CSV output to a configurable storage adapter to avoid high server RAM usage.
Changes:
- Introduces
exportBigDatasetplugin options (buffer size, read chunk size, storage adapter) and wires background-jobs task registration. - Adds new server endpoints to start an export job and retrieve a presigned download URL.
- Adds UI components to start background exports and to display/download finished export jobs.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| types.ts | Adds exportBigDataset options and storage adapter typing. |
| index.ts | Registers background export job handler + adds endpoints for starting exports and fetching download links; passes UI meta. |
| exportMultipartUpload.ts | Implements streamed CSV generation + multipart upload export job runner and download URL retrieval. |
| custom/ExportCsv.vue | Switches export action to background job when enabled; reuses previously fetched counts for progress estimates. |
| custom/ExportCsvJobViewComponent.vue | Adds a job details UI panel with download action for completed exports. |
| package.json | Adds csv dependency and @adminforth/background-jobs dev dependency; bumps dev adminforth. |
| pnpm-workspace.yaml | Allows building classic-level (dependency of background jobs). |
| pnpm-lock.yaml | Locks new dependencies and updates adminforth version. |
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
Comments suppressed due to low confidence (1)
types.ts:13
- Typo in docs: “more that” → “more than” (and minor units/wording).
/**
* Size of the buffer, where records will be stored before upload. Min size is 5 MB. The default value is 5 MB. (takes RAM memory on the server)
* AWS S3 doesn't support more that 10000 parts per upload, so if you have a huge database you want to export (for default settings more that 100 Gb) - increase buffer size
*/
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -1,5 +1,24 @@ | |||
| import {type PluginsCommonOptions } from "adminforth"; | |||
| import {type PluginsCommonOptions, StorageAdapter } from "adminforth"; | |||
Comment on lines
+4
to
+8
| /** | ||
| * If you are going to export a huge dataset - it can make your server run out of memory, because all exported records are stored in RAM | ||
| * To prevent this, you can use exportBigDataset option. | ||
| * It requires a background jobs plugin to be setted up | ||
| */ |
Comment on lines
42
to
44
| auditLogPlugin: Record<string, any> | undefined; | ||
|
|
||
| backgroundJobsPlugin: any; | ||
|
|
Comment on lines
+101
to
+107
| function getJobsResourceMeta(backgroundJobsPlugin: BackgroundJobsPlugin): { resourceId: string; pkColumn: string } { | ||
| const resourceConfig = backgroundJobsPlugin.resourceConfig; | ||
| return { | ||
| resourceId: resourceConfig.resourceId, | ||
| pkColumn: resourceConfig.columns.find((col) => col.primaryKey).name, | ||
| }; | ||
| } |
Comment on lines
+43
to
+50
| // numbers and booleans are left bare, everything else is quoted: | ||
| // quoting all columns breaks BI/Excel tasks | ||
| columnsToForceQuote: columns.map((col) => ( | ||
| col.type !== AdminForthDataTypes.FLOAT | ||
| && col.type !== AdminForthDataTypes.INTEGER | ||
| && col.type !== AdminForthDataTypes.BOOLEAN | ||
| && col.type !== AdminForthDataTypes.DECIMAL | ||
| )), |
Comment on lines
+194
to
+196
| // BOM keeps Excel happy with non-ASCII values | ||
| // Add this symbol to the beginning of the file to indicate that it is UTF-8 encoded. (requred for some versions of Excel for some reason. Without this symbol encoding can be broken) | ||
| await writer.write('' + buildCsvChunk([fields], columnsToForceQuote)); |
Comment on lines
27
to
34
| "peerDependencies": { | ||
| "adminforth": "^3.8.2" | ||
| }, | ||
| "devDependencies": { | ||
| "@adminforth/background-jobs": "^1.18.4", | ||
| "@types/node": "^22.10.7", | ||
| "adminforth": "^3.8.2", | ||
| "adminforth": "^3.12.1", | ||
| "semantic-release": "^24.2.1", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.