Report and ban users in a Flutter app with Firebase
A report button takes an afternoon. Making reports trustworthy and bans actually stick takes a few backend decisions. Here is an architecture that works with Firebase, with code for Flutter, Cloud Functions and Firestore rules.
This guide shows how a Flutter app with Firebase lets users report content and block each other, and how you suspend accounts. It works without Limenia. Where a moderation tool saves you work, the last section says so.
The architecture: app → Cloud Function → moderation → back
Four parts, each with one job:
- Flutter app: shows the report button and the block option. It sends only IDs and the reason.
- Cloud Function (callable): accepts the report, takes the reporter from the ID token, reads the reported content from Firestore itself and stores a snapshot.
- Moderation: someone reviews the report and decides. That can be your own admin screen or a tool connected through an API.
- Back into Firebase: a function applies the decision: it hides the content, suspends the account and sends the statement of reasons.
Flutter app ──httpsCallable──▶ Cloud Function ──▶ Moderation
│ decision
Firestore / Firebase Auth ◀── Cloud Function ◀────────┘
Why not write straight from the app into a reports collection? Because then the app decides what was reported and by whom. A modified client can submit someone else’s text as the snapshot. A function reads the content from your database, so your database stays the source of truth.
The report button in Flutter
The app only asks for the reason and calls the function. For notices about illegal content, Art. 16(2) DSA requires, among other things, a sufficiently substantiated explanation. So offer a free-text field, at least for an “illegal content” reason.
import 'package:cloud_functions/cloud_functions.dart';
import 'package:flutter/material.dart';
const reasons = {
'spam': 'Spam',
'harassment_hate': 'Harassment or hate',
'sexual_content': 'Sexual content',
'other': 'Something else',
};
Future<void> showReportSheet(BuildContext context, String commentId) async {
final reason = await showModalBottomSheet<String>(
context: context,
builder: (context) => SafeArea(
child: ListView(
shrinkWrap: true,
children: [
for (final entry in reasons.entries)
ListTile(
title: Text(entry.value),
onTap: () => Navigator.pop(context, entry.key),
),
],
),
),
);
if (reason == null) return;
final callable = FirebaseFunctions.instanceFor(region: 'europe-west3')
.httpsCallable('reportComment');
try {
await callable.call(<String, dynamic>{'commentId': commentId, 'reasonCategory': reason});
if (context.mounted) {
ScaffoldMessenger.of(context).showSnackBar(
const SnackBar(content: Text('Thanks. We will review this report.')),
);
}
} on FirebaseFunctionsException catch (e) {
debugPrint('report failed: ${e.code}'); // log the code only, never content
}
}
The function takes the reporter from request.auth, never from request.data:
import { initializeApp } from "firebase-admin/app";
import { getAuth } from "firebase-admin/auth";
import { FieldValue, getFirestore } from "firebase-admin/firestore";
import { HttpsError, onCall } from "firebase-functions/v2/https";
initializeApp();
export const reportComment = onCall({ region: "europe-west3" }, async (request) => {
if (!request.auth) throw new HttpsError("unauthenticated", "Sign in first.");
const { commentId, reasonCategory, reasonText } = request.data ?? {};
if (typeof commentId !== "string" || typeof reasonCategory !== "string") {
throw new HttpsError("invalid-argument", "commentId and reasonCategory are required.");
}
const db = getFirestore();
const comment = await db.collection("comments").doc(commentId).get();
if (!comment.exists) throw new HttpsError("not-found", "Comment not found.");
const reporterUid = request.auth.uid; // from the ID token, never from request.data
await db.collection("reports").doc(`${commentId}_${reporterUid}`).set({
reporterUid,
commentId,
authorUid: comment.get("authorUid"),
text: comment.get("text"), // snapshot: the comment may be edited or deleted later
reasonCategory,
reasonText: typeof reasonText === "string" ? reasonText.slice(0, 2000) : null,
status: "open",
createdAt: FieldValue.serverTimestamp(),
});
return { ok: true };
});
Building the document ID from the comment and the reporter makes retries harmless: tapping twice creates one report, not two. Lock the reports collection for clients entirely in your Firestore rules; only the Admin SDK writes there.
Blocking between users
Blocking is not the same as banning. One user hides another for themselves and stops them from sending messages. The account stays active, and you as the provider decide nothing.
The stores ask for this in different ways. Google Play requires an in-app blocking feature where users can interact one-to-one (for example direct messages, tagging, mentions). Apple’s guideline 1.2 lists “the ability to block abusive users from the service”. More in the guide on App Store Guideline 1.2.
A block list per user is enough: users/{uid}/blocked/{otherUid}. The app uses it to filter feeds and comments on the device. Enforce the block in your rules, though, or a modified client can still send messages:
rules_version = '2';
service cloud.firestore {
match /databases/{database}/documents {
function signedIn() { return request.auth != null; }
function notBanned() {
return !exists(/databases/$(database)/documents/bans/$(request.auth.uid));
}
function members(chatId) {
return get(/databases/$(database)/documents/chats/$(chatId)).data.members;
}
function hasBlocked(owner, other) {
return exists(/databases/$(database)/documents/users/$(owner)/blocked/$(other));
}
match /users/{uid}/blocked/{otherUid} {
allow read, write: if signedIn() && request.auth.uid == uid;
}
match /chats/{chatId}/messages/{messageId} {
allow create: if signedIn() && notBanned()
&& request.resource.data.senderUid == request.auth.uid
&& request.auth.uid in members(chatId)
&& request.resource.data.recipientUid in members(chatId)
&& !hasBlocked(request.resource.data.recipientUid, request.auth.uid);
}
}
}
Every get() and exists() in your rules is billed as a document read. Keep these checks on write rules rather than on every list read.
Suspending an account: updateUser({disabled: true}) and revoking refresh tokens
You suspend an account with the Admin SDK, so in a function or another backend. Two calls belong together:
import { getAuth } from "firebase-admin/auth";
import { FieldValue, getFirestore, Timestamp } from "firebase-admin/firestore";
export async function suspendUser(uid, until /* Date, or null for a permanent ban */, decisionId) {
await getAuth().updateUser(uid, { disabled: true }); // no new sign-in, no token refresh
await getAuth().revokeRefreshTokens(uid); // existing sessions end at the next refresh
await getFirestore().collection("bans").doc(uid).set({
until: until ? Timestamp.fromDate(until) : null,
decisionId,
createdAt: FieldValue.serverTimestamp(),
});
}
Firebase Authentication has no ban with an expiry date. For a temporary suspension you lift it yourself, for example every hour:
import { onSchedule } from "firebase-functions/v2/scheduler";
export const liftExpiredSuspensions = onSchedule(
{ schedule: "every 60 minutes", region: "europe-west3" },
async () => {
const due = await getFirestore().collection("bans").where("until", "<=", Timestamp.now()).get();
for (const doc of due.docs) {
await getAuth().updateUser(doc.id, { disabled: false });
await doc.ref.delete();
}
},
);
Permanent bans have until: null and drop out of the query. Do not delete banned accounts: if a complaint succeeds, you need to be able to lift the ban.
What about custom claims?
A claim like banned: true only takes effect once the client gets a new ID token, after a new sign-in or the next refresh. A disabled account gets no new token at all. Firebase also says claims are meant for access control only, and limits them to 1000 bytes. They are good for roles like “moderator”; for bans, disabled plus a ban list is the better fit.
Why banned users can keep going for up to an hour
Firebase ID tokens are valid for one hour. Disabling and revoking only stop new tokens from being issued. Firestore and Storage rules keep accepting a token that was already issued until it expires. onCall also verifies the token without a revocation check.
Where the ban must apply at once, you have three levers:
-
Your own HTTP endpoints:
getAuth().verifyIdToken(token, true). WithcheckRevoked, the Admin SDK also asks the Auth backend and throwsauth/user-disabledorauth/id-token-revoked. -
Callable functions: the raw token is in
request.auth.rawToken. Check it again for sensitive actions:try { await getAuth().verifyIdToken(request.auth.rawToken, true); } catch { throw new HttpsError("permission-denied", "Account suspended."); } -
Firestore rules: the
bans/{uid}list from above, checked withnotBanned()in every write rule for user content.
Firebase documents a similar pattern for the Realtime Database, with a stored revocation time compared to auth_time in the token. A ban list is simpler and enough for moderation.
Removing content and showing the statement of reasons
Removing usually means hiding rather than deleting, for example with status: "removed" and a read rule that only returns visible content. That way you can restore it after a successful complaint. Set a period after which removed content is deleted for good.
Under Art. 17 DSA, every suspension and every removal needs a clear and specific statement of reasons, at the latest when the measure takes effect. It names the measure and its duration, the facts, whether automated means were used, the rule or law that was breached, and the available redress. The guide on the statement of reasons under Art. 17 DSA covers the details.
An account suspension has one catch: the person can no longer sign in. Send the statement by e-mail and show it at the sign-in attempt as well. FlutterFire then throws a FirebaseAuthException with the code user-disabled. Catch it and show a screen that explains the suspension and how to complain, instead of a generic error.
Don’t forget the reporters
Art. 16(5) DSA requires you to tell the person who reported content about your decision. Store who sent each report and notify them once the case is decided.
With Limenia: webhook or Firebase connector
Limenia can take over the “moderation” part of the architecture: grouping reports, deciding in the dashboard, writing the Art. 17 statement of reasons, informing reporters and accepting complaints. Your reportComment function then forwards to POST /v1/reports instead of storing the report itself. The API key lives as a secret in your functions (firebase functions:secrets:set LIMENIA_API_KEY), never in the app.
Decisions come back in two ways:
- Webhook
decision.created: anonRequestfunction checks the signature overreq.rawBodyand applies the action: hide the content, update the ban list and send the ready-made statement of reasons to the affected user. - Firebase connector (Pro and Premium): Limenia disables the account in Firebase Authentication itself and revokes its refresh tokens; when a ban is lifted or a suspension ends, it enables the account again. Limenia’s service account gets a custom role with three permissions, and Limenia stores no keys of your project. Your backend has to send the Firebase UID as the user ID.
The connector does not touch Firestore, Storage or custom claims. You still need the webhook, and checking with verifyIdToken(token, true) still makes sense. A working Cloud Functions example is in the examples repository on GitHub, and section 2.4 of the integration guide walks through every step. For the app there is the limenia package on pub.dev with a ready-made report sheet, complaint form, statement of reasons and restriction notice in package:limenia/ui.dart; the report button from above works just as well. More on the connector in Moderation for Firebase apps and on the app side in Moderation for Flutter apps.
Firebase and Flutter are trademarks of Google LLC. Apple and App Store are trademarks of Apple Inc. Google Play is a trademark of Google LLC.
Sources
- Firebase: Manage User Sessions (revokeRefreshTokens, checkRevoked)firebase.google.com
- Firebase: Manage Users (Admin SDK)firebase.google.com
- Firebase: Control Access with Custom Claimsfirebase.google.com
- Firebase: Call functions from your app (callable functions)firebase.google.com
- Firebase: Schedule functionsfirebase.google.com
- Firebase: Writing conditions for Cloud Firestore Security Rulesfirebase.google.com
- Regulation (EU) 2022/2065 (Digital Services Act), Art. 16 and 17, EUR-Lexeur-lex.europa.eu
- Apple App Review Guidelines, 1.2 User-Generated Contentdeveloper.apple.com
- Google Play Console Help: User Generated Contentsupport.google.com
- Limenia integration guide (section 2.4 Flutter + Firebase)limenia.eu
- Limenia example backends on GitHub (Apache 2.0)github.com