Custom error handling with AuthJS (Next-Auth Beta) and a Custom Backend in Next.js



This content originally appeared on Level Up Coding – Medium and was authored by Nnebedum Favour Ifechukwu

Custom error handling for NextAuth and backend API in Next.js authentication flow

This is a practical guide (based on personal experience) to integrating your backend API with AuthJS (NextAuth), covering both credential login and Google OAuth, with tips on token handling and backend verification.

We’ll cover:
✅ Setting up email/password login using the Credentials Provider
✅ Using the Google OAuth Provider alongside a backend validation flow
✅ Handling tokens, user sessions, and backend errors cleanly
✅ Protecting routes and wiring it all together

Whether you’re building an internal tool or a production app, this pattern gives you full control over authentication without sacrificing the built-in power of AuthJS.

Credentials Provider (Email & Password)

For this setup I am assuming your backend has a /auth/sign-in endpoint that:
– Accepts email and password via POST
– Returns a user object and an access token on success
– Returns custom errors for client issues (e.g. wrong password) or server errors

Here’s a simplified server action that calls the backend:

// src/lib/actions/auth.actions.ts
"use server"

export const signIn = async (payload: { email: string; password: string }) => {
try {
const response = await baseUrl.post("/auth/sign-in", payload);
const { token, data: userData } = response?.data?.data;

if (token && userData) {
const formattedData = {
token,
data: userData as IUser, // type assertion to keep NextAuth happy
};
return { data: formattedData };
}
} catch (error) {
return { error: getError(error) };
}
};

🛠 getError is a small utility you can use to cleanly extract messages from Axios, fetch, or native JS errors.

🛠 baseUrl is my axios instance.

Google OAuth Provider (With Backend Verification)

After a successful sign-in with Google via AuthJS, we take the Google token and verify it with our backend.
Here’s how the flow works:
– AuthJS handles the initial Google sign-in.
– We extract the Google ID token from the user profile.
– We send that token to our backend to verify it.
– Backend validates it via Google’s API, then:
✅ If the email exists: returns the user and session token.
🆕 If not: creates the user and returns the session token.

Here’s my server action for that backend call:

// src/lib/actions/auth.actions.ts
"use server"

export const googleLogin = async (payload: any) => {
try {
const response = await baseUrl.post("/auth/google/login", payload);
const { token, data: userData } = response?.data?.data;

if (token && userData) {
const formattedData = {
token,
data: userData as IUser,
};
return { data: formattedData };
}
} catch (error) {
throw new Error(getError(error));
}
};

With this setup in place, we’re ready to move into the AuthJS configuration and show how to wire everything up in your Next.js project: providers, callbacks, middleware, and client-side usage.

Setting Up AuthJS with Credentials and Google Providers

Here’s how we configure AuthJS (NextAuth v5) to work with both a custom email/password backend and Google OAuth, while still keeping full control of error handling, token flow, and user session.

We’ll be working with a shared auth.ts file and exporting everything from there — as suggested by NextAuth.

// src/auth.ts
import NextAuth, { AuthError } from "next-auth";
import Google from "next-auth/providers/google";
import CredentialsProvider from "next-auth/providers/credentials";

import { googleLogin, signIn as Login } from "@/lib/actions/auth.actions";
import { getError } from "./lib";

/*
NextAuth suppresses error messages by default.
To get around this, we create a custom AuthError class that lets us pass clean,
actionable error messages back to the frontend:
*/
export class InvalidLoginError extends AuthError {
code = "custom";
errorMessage: string;
constructor(message?: any, errorOptions?: any) {
super(message, errorOptions);
this.errorMessage = message;
}
}

export const { handlers, signIn, signOut, auth } = NextAuth({
pages: {
signIn: "/signin",
error: "/signin",
},
session: {
strategy: "jwt",
maxAge: 24 * 60 * 60, // 1 day
},
jwt: {
maxAge: 30 * 24 * 60 * 60, // 30 days
},
providers: [
// Google Provider
Google({
clientId: process.env.AUTH_GOOGLE_ID,
clientSecret: process.env.AUTH_GOOGLE_SECRET,
authorization: {
params: {
prompt: "consent",
access_type: "offline",
response_type: "code",
},
},
}),

// Credentials Provider
CredentialsProvider({
name: "Credentials",
credentials: {
email: { label: "Email", type: "text" },
password: { label: "Password", type: "password" },
},

async authorize(credentials) {
const email = credentials?.email as string;
const password = credentials?.password as string;

try {
if (!password || !email) {
throw new InvalidLoginError("Invalid credentials");
}

const response = await Login({ email, password });

if (response?.error) {
throw new Error(response?.error);
}

return response?.data;
} catch (error: any) {
throw new InvalidLoginError(getError(error));
}
},
}),
],
callbacks: {
authorized: async ({ auth }) => {
return !!auth; // Only logged-in users are authorized
},

async signIn({ user, account }) {
if (account?.provider === "google") {
try {
const res = await googleLogin({
clientId: process.env.AUTH_GOOGLE_ID!,
select_by: "random string", // or anything required
credential: account.id_token!,
});

if (!res?.token) {
throw new InvalidLoginError("Failed to login with Google");
}

(user as any).token = res.token;
(user as any).data = res.data;

return true;
} catch (error) {
throw new InvalidLoginError(getError(error));
}
}

return true; // For credentials
},

async jwt({ token, user }) {
if ((user as any)?.token) {
token.token = (user as any)?.token;
token.data = (user as any)?.data;
}
return token;
},

async session({ session, token }) {
session.token = token.token as string;
session.user = token.data as any;
return session;
},
}
});

API Route + Middleware Setup

Instead of duplicating logic in your API route and middleware, you can just reference the shared auth export from above.

// src/app/api/auth/[...nextauth]/route.ts
import { handlers } from "@/auth";

export const { GET, POST } = handlers;
// src/middleware.ts
export { auth as middleware } from "@/auth";

export const config = {
matcher: ["/", "/(protected-page-1|protected-page-2)/:path*"],
};

Building a Custom Login Form (Email & Password)

For the credentials login, we create a custom form that:
– Uses controlled inputs (useState)
– Handles async login through a server action wrapper (authenticate)
– Displays toasts for feedback
– Redirects the user on success using router.replace

Here’s a breakdown of the full login component:

"use client";

import toast from "react-hot-toast";
import { SignInOptions } from "next-auth/react";
import { useSearchParams, useRouter } from "next/navigation";
import { useState } from "react";
import Link from "next/link";
import { Input, SubmitButton } from "../atoms"; // Reusable input/button components
import { getError } from "@/lib";
import { authenticate } from "@/lib/actions/auth.actions";

export const LoginForm = () => {
const router = useRouter();
const [email, setEmail] = useState("");
const [password, setPassword] = useState("");

const searchParams = useSearchParams();
const callbackUrl = searchParams.get("callbackUrl") || "/";

const formAction = async () => {
toast.dismiss();
try {
if (!email || !password) {
throw new Error("Fill all required fields!");
}

const credentials: SignInOptions<false> = {
redirect: false,
email,
password,
redirectTo: callbackUrl,
};

const res = await authenticate("credentials", credentials);

if (res?.error) {
throw new Error(res.error);
}

toast.success("Login successful, redirecting...");
return router.replace(`${res?.url || "/"}`);
} catch (error: any) {
toast.error(getError(error) || "Login failed! Please try again.");
}
};

return (
<form
action={formAction}
className="flex flex-col w-full gap-4 pt-6 mx-auto text-sm text-black lg:mt-auto font-body"
>
<div className="flex flex-col w-full gap-4 max-w-[500px]">
<Input
name="email"
type="email"
id="email"
label="Email Address"
value={email}
onChange={(e) => setEmail(e.target.value)}
placeholder="user@example.com"
/>

<Input
name="password"
id="password"
type="password"
label="Password"
value={password}
onChange={(e) => setPassword(e.target.value)}
placeholder="**********"
/>

<Link
href="/forgotpassword"
className="ml-auto font-medium text-dark w-fit hover:underline"
>
Forgot password?
</Link>
</div>

<SubmitButton
className="rounded-[20px] min-w-full md:min-w-fit max-w-[201px] w-full text-center justify-center my-4"
>
Sign In
</SubmitButton>

<div className="flex gap-1">
<p className="text-[#64748B]">Don&apos;t have an account? </p>
<Link
href="/signup"
className="font-semibold text-primary hover:underline"
>
Sign up
</Link>
</div>
</form>
);
};

🧪 Auth Helper: Handling Errors with authenticate()

NextAuth by default swallows detailed errors from the backend when using the Credentials Provider. This makes it hard to show helpful messages like:

❌ “Incorrect email or password”

Instead, we wrap the signIn() call in a small authenticate() helper function that catches custom InvalidLoginError instances (like we defined earlier) and extracts a clean message using our getError() utility.

// src/lib/actions/auth.actions.ts
"use server"
import { SignInOptions, signIn as NextAuthSignIn } from "next-auth/react";
import { getError } from "@/lib";

/*
This small wrapper:
Lets you surface errors thrown by your authorize() logic
(e.g. InvalidLoginError)
*/

export async function authenticate(
provider: string,
signinOptions: SignInOptions<false> = {}
) {
try {
return await NextAuthSignIn(provider, signinOptions);
} catch (error: any) {
return {
error: error?.errorMessage || getError(error),
};
}
}

Auth Helper: Handling Errors with authenticate()

As mentioned earlier, we wrap signIn() with a custom authenticate() helper to capture backend errors and cleanly show them on the frontend.

But here’s a subtle detail that trips up a lot of people 👇

Which signIn are we using?

When logging in with credentials, we don’t use the default signIn() from "next-auth/react".

Instead:

  • ✅ We use the signIn exported from our auth.ts setup file (i.e. NextAuth().signIn)
  • ✅ This supports catching and returning custom errors like InvalidLoginError

For other providers like Google, we fall back to the built-in signIn() from next-auth/reactalthough personally I have never experienced any errors with the OAUTH providers.

Now that sign-in is set up, let’s see how to access session data across your app.

Accessing the Session (Server & Client)

Once users are authenticated, you can access their session from anywhere in your Next.js App Router project.

  • On the Server (e.g. pages, layouts, or server actions)
// app/dashboard/page.tsx or in any server component
import { auth } from "@/auth";

const session = await auth();

console.log(session?.user?.email);

This is useful for:

  • Protecting routes
  • Rendering dynamic UI (e.g. based on roles or permissions)
  • Fetching user-specific data
  • In Client Components
"use client";
import { useSession } from "next-auth/react";

export const Dashboard = () => {
const { data: session } = useSession();

return <p>Hello, {session?.user?.name}</p>;
};

This is perfect for:

  • Header components (showing user info)
  • Buttons like “Logout”
  • Personalizing user-facing UI

Logging Out

To log out, you use the built-in signOut() function:

"use client"

import { signOut } from "next-auth/react";

<button onClick={() => signOut({ callbackUrl: "/signin" })}>
Logout
</button>

So yea, that’s how I would set up and use NextAuth in a Next.js project with custom backend support.

We covered:

  • Credentials sign-in with email and password
  • Google login and how I pass the token to the backend for verification
  • How I return proper errors using InvalidLoginError
  • A small helper (authenticate) to cleanly handle those errors in the UI
  • How to access sessions on both server and client
  • Protecting routes with middleware
  • Logging out with a simple signOut() call

Hope this helps someone out there trying to spin up a similar flow.

Got questions? Feedback? I’d love to hear it.

Also, I’m curious — how do you handle authentication in your React or Next.js projects?

Drop a comment, and let’s share some ideas 👇

🔗 Helpful Links

Here are some useful links if you want to dive deeper or need docs for anything I mentioned:


Custom error handling with AuthJS (Next-Auth Beta) and a Custom Backend in Next.js was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.


This content originally appeared on Level Up Coding – Medium and was authored by Nnebedum Favour Ifechukwu