How to Handle Large File Uploads (Without Losing Your Mind)

Why Consider Optimizing Large File Uploads In frontend development, we often encounter scenarios where file uploads are required, such as uploading images, videos, audio files, etc. During file uploads, we may encounter issues if the files being too large, such as: Leading to long upload times and poor user experience. Causing excessive server load and resource consumption. Resulting in upload failures in unstable network conditions, requiring re-uploads and wasting time and bandwidth. Leading to high browser memory usage, affecting performance and stability. To address these problems, we need to optimize large file uploads. In modern internet applications, users have increasingly high demands for file uploads, such as: Sharing high-resolution images and videos on social platforms. Submitting assignments and course materials on educational platforms. Uploading project documents and reports on enterprise platforms. In these scenarios, the files users need to upload are often quite large, sometimes reaching hundreds of megabytes or even several gigabytes. If we use traditional file upload methods, we will face the aforementioned issues. The traditional file upload method involves sending the entire file as a single request body to the server. This approach has several drawbacks: Long upload times: Due to the large file size, it takes a long time to transfer the data, requiring users to wait a long time to see the upload results. High server load: Large files require the server to receive and process large amounts of data at once, potentially consuming excessive server memory, CPU, bandwidth, and other resources. Prone to failure in unstable networks: Large file transfers are susceptible to network issues such as disconnections, timeouts, or packet loss, leading to upload failures and forcing users to re-upload the entire file. High browser memory usage: Large files require the browser to read the entire file into memory and maintain the connection, which may result in high memory usage and impact the performance of other pages. To address these problems, optimizing large file uploads is essential. Design Approach The primary approaches to optimizing large file uploads include: 1. Chunking Divide a large file into smaller chunks, with each chunk sent as a separate request to the server. This reduces the amount of data per request, shortens upload times, decreases server load, and enables resumable uploads. function sliceFile(file, chunkSize) { const fileSize = file.size; const chunks = Math.ceil(fileSize / chunkSize); const slices = Array.from({ length: chunks }, (_, index) => { const start = index * chunkSize; const end = start + chunkSize; return file.slice(start, end); }); return slices; } 2. Concurrency Send multiple chunk requests to the server simultaneously to fully utilize network bandwidth and server resources, improving user experience. async function uploadChunks(fileChunks) { const uploadPromises = fileChunks.map((chunk) => fetch('/upload', { method: 'POST', body: chunk }) ); const responses = await Promise.all(uploadPromises); return responses; } 3. Compression Compress each chunk before sending it to the server to further reduce data size and improve transfer efficiency. async function compressChunk(chunk) { const compressedChunk = await new Promise((resolve, reject) => { const reader = new FileReader(); reader.onload = (event) => { const result = pako.deflate(event.target.result); resolve(result); }; reader.onerror = (event) => reject(event.error); reader.readAsArrayBuffer(chunk); }); return compressedChunk; } 4. Validation Validate each chunk before or after transmission to ensure data integrity and correctness, avoiding redundant or erroneous data transfers. async function verifyChunk(chunk) { const hash = await calculateHash(chunk); const response = await fetch(`/verify?hash=${hash}`); const result = await response.json(); return result; } 5. Resumable Uploads In the event of a network failure, continue uploading from the point of interruption instead of starting over, saving time and improving speed. async function resumeUpload(file, resumeByte) { const blob = file.slice(resumeByte); const formData = new FormData(); formData.append('file', blob); const response = await fetch('/upload', { method: 'POST', body: formData }); const result = await response.json(); return result; } 6. Instant Uploads Before slicing the file for upload, calculate its hash and send it to the server. If the server finds an identical file, it can immediately return a success response, avoiding redundant uploads. async function checkFileExists(file) { const hash = await calculateHash(file); const response = await fetch(`/check?hash=${hash}`); const result = await resp

function sliceFile(file, chunkSize) {
  const fileSize = file.size;
  const chunks = Math.ceil(fileSize / chunkSize);
  const slices = Array.from({ length: chunks }, (_, index) => {
    const start = index * chunkSize;
    const end = start + chunkSize;
    return file.slice(start, end);
  });
  return slices;
}

async function uploadChunks(fileChunks) {
  const uploadPromises = fileChunks.map((chunk) =>
    fetch('/upload', { method: 'POST', body: chunk })
  );
  const responses = await Promise.all(uploadPromises);
  return responses;
}

async function compressChunk(chunk) {
  const compressedChunk = await new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.onload = (event) => {
      const result = pako.deflate(event.target.result);
      resolve(result);
    };
    reader.onerror = (event) => reject(event.error);
    reader.readAsArrayBuffer(chunk);
  });
  return compressedChunk;
}

async function verifyChunk(chunk) {
  const hash = await calculateHash(chunk);
  const response = await fetch(`/verify?hash=${hash}`);
  const result = await response.json();
  return result;
}

async function resumeUpload(file, resumeByte) {
  const blob = file.slice(resumeByte);
  const formData = new FormData();
  formData.append('file', blob);
  const response = await fetch('/upload', { method: 'POST', body: formData });
  const result = await response.json();
  return result;
}

async function checkFileExists(file) {
  const hash = await calculateHash(file);
  const response = await fetch(`/check?hash=${hash}`);
  const result = await response.json();
  return result;
}

Related Posts

Use Power Query SDK in Visual Studio Code for Power BI

Jan 15, 2025

What's the point of "use case optimal query" and partia...

Jan 18, 2025

Item 84: Não dependa do agendador de threads

Jan 16, 2025