File Ingest Instability

Introduction

File ingest instability can happen when uploaded files are too large for the ingest process to handle reliably. In Kleene, large files can make file ingest slower, more memory-intensive, or less stable. Splitting a large file into smaller parts is often the safest way to complete the ingest.

This article explains when file size may be causing ingest issues and how to reduce the risk.

Issue description

A file ingest may fail, stall, or behave unpredictably when the file is very large. Anecdotally, files around 100-150 MB tend to be safer, while files closer to 250 MB or above can become unstable depending on row count, structure and warehouse load behaviour.

Very large row counts can also contribute to instability. For example, a multi-million-row file may need to be split before ingesting successfully.

Signs

You may be dealing with this issue if:

• A file ingest fails without an obvious schema or credential problem.
• The file is large, especially above 150–250 MB.
• The file contains millions of rows.
• The ingest works after the file is split into smaller chunks.
• Smaller test files load successfully.
• The issue appears during upload, parsing, or warehouse loading.

Basic troubleshooting steps

Start with the following checks:

• Check the file size.
• Check the number of rows.
• Confirm the file format is valid.
• Test with a smaller sample file.
• Split the large file into smaller files.
• Re-run the ingest using the smaller chunks.
• Confirm each split file lands and loads successfully.
• Review whether the file can be compressed or partitioned before upload.

Common causes and how to fix them

File is too large for stable ingest

Large files can put more strain on upload, parsing, staging, and warehouse load steps. Even if the file is technically valid, it may be more likely to fail.

How to fix it: split the file into smaller files before ingesting. Aim for safer file sizes, such as around 100–150 MB where practical.

Too many rows in one file

A file with millions of rows can be difficult to process in a single ingest. The issue may be related to the amount of data rather than a bad row.

How to fix it: split the file by row count. Keep headers consistent across the split files where required.

File works as a sample but not at full size

A smaller sample may ingest correctly while the full file fails. This suggests the setup is mostly correct, but the full file is too large or contains problematic rows later in the file.

How to fix it: split the file and test each part. If only one split fails, inspect that chunk for schema, delimiter, encoding, or bad-row issues.

Warehouse load pressure

Large files can take longer to copy into the destination warehouse and may be more exposed to load timeouts or staging instability.

How to fix it: reduce file size, split data by period or logical partition, and load in smaller batches.

Practical troubleshooting workflow

1. Check the file size and row count.
2. Confirm whether the file is above the normal safe range.
3. Create a smaller sample and test whether it ingests successfully.
4. If the sample succeeds, split the full file into smaller chunks.
5. Keep column order, headers, delimiter, and encoding consistent.
6. Ingest the split files one at a time or as supported by the workflow.
7. Confirm that each file lands and loads successfully.
8. If one chunk fails, inspect that chunk for malformed rows or schema differences.
9. If all chunks succeed, document that the original issue was file size or ingest stability.

Best practices to avoid file ingest instability

• Keep file sizes around 100–150 MB where possible.
• Split files that approach 250 MB or contain millions of rows.
• Use consistent headers and column order across split files.
• Validate delimiter, encoding, and quoting before upload.
• Test with a small sample before loading a very large file.
• Partition large exports by date or logical grouping where possible.
• Avoid treating a large-file failure as a credential issue unless smaller files also fail.

Additional information

If a large file ingest is unstable, splitting the file is usually the fastest workaround. For example, a file with several million rows may need to be divided into smaller parts before loading reliably.

If split files still fail, investigate file structure, malformed rows, column count mismatch, encoding, and destination schema.