Payload validation

This content originally appeared on DEV Community and was authored by Marcelo Magario

Recently I faced a problem at work: our mailing system stopped working and a heap of emails weren’t being sent. When we looked at RabbitMQ:

There was one “Unacked” message stuck on the consumer, and lots of “Ready” messages were piling up.

After investigating, we found that our mailer supplier’s API broke because the mailing payload was over 30 MB. Digging deeper, we saw that one user had sent a mailing with a 70 MB attachment.

First, I tried adding attachment-file-size validation and overall payload-size validation (body + attachments) on our backend. But the problem was that the user only saw “the file is too large” once they tried to save or send the mailing. So, after analyzing our process, I ended up putting validation in both the frontend and backend.

Frontend:

Validating the attachment size on the frontend lets the user know immediately if the file is too large:

const handleAttachment = () => {
  if (selectedFiles && selectedFiles[0]) {
    const sizeInMB = selectedFiles[0].size / (1024 * 1024)
    if (sizeInMB > 9) {
      setShowSizeError(true)
      return
    }
  }
  closeModal()
}

Backend

To make sure our mailer supplier’s API won’t break again, I added payload validation on our backend. It calculates the size of all attachments plus the email content, then checks whether the total size exceeds our limit:

// helper/requestSize.js
module.exports.computeRequestSize = (request) => {
  let totalSize = 0
  let bodyBytes = 0
  let attachmentsBytes = 0

  if (request.body) {
    const serialized = JSON.stringify(request.body)
    bodyBytes = Buffer.byteLength(serialized, 'utf8')
    totalSize += bodyBytes
  }

  if (request.files) {
    Object.values(request.files).forEach(file => {
      attachmentsBytes += file.size
    })
    totalSize += attachmentsBytes
  }

  return totalSize
}

and on the endpoint:

const payloadSize = util.computeRequestSize(request)
if (payloadSize > CONFIG.MAX_REQUEST_SIZE) {
  response.sendError400(CONFIG.ERROR_PAYLOAD_TOO_LARGE)
  return next()
}

For safety reasons I always change a little bit my code before post here.

Now we know this won’t happen again.
What would you do?

This content originally appeared on DEV Community and was authored by Marcelo Magario