Selective protection against bots and scraping

Protect access.
Preserve openness.

Public search portals must remain available to people while continuing to serve APIs, OAI-PMH and trusted academic services. This implementation applies a proportional response to each type of traffic before it reaches the application.

See how it works Review the evidence

Validated implementation · May–July 2026 · LA Referencia IOI

The access problem

Open services also need operational protection.

Not all automated traffic has the same purpose. Search bots, academic harvesters, monitoring systems and human visitors compete for the same infrastructure but require different treatment.

Costly dynamic routes

Search and record pages require processing and can be saturated by large volumes of concurrent requests.

Legitimate automation

OAI-PMH, APIs, monitors and recognised academic aggregators must remain available for the network to function.

Shared public capacity

Every intensive request consumes resources that should remain available for people and authorised scholarly services.

Implemented architecture

One entry point.
Three possible decisions.

Nginx Proxy Manager manages HTTPS, hosts and trusted networks. Anubis evaluates an ordered YAML policy and allows, rejects or challenges each request before forwarding it to the final service.

01InternetHTTPS requests

→

02Nginx Proxy ManagerTLS, hosts, headers and trusted networks

→

03AnubisOrdered policy and computational challenge

→

04Final serviceVuFind · OJS · WordPress · Omeka

ALLOW

Keep scholarly services open

OAI-PMH, basic resources, monitoring and explicitly trusted services can bypass the challenge.

/OAI · /.well-known · robots.txt

DENY

Stop identified scrapers

Known intensive agents are rejected by an explicit policy rule and receive a traceable HTTP 403 response.

GPTBot · Amazonbot · ClaudeBot

CHALLENGE

Verify conventional browsers

A browser completes a computational challenge and receives a signed cookie before reaching the portal.

User-Agent: Mozilla

Scope: Anubis mitigates bots and intensive scraping. Application security and other attack families remain the responsibility of the proxy, server, application and complementary controls.

Observed evidence

One hour made the pressure visible.

On 30 May 2026, one hour of logs from an Internet-facing VuFind instance protected by Anubis was analysed. The figures below describe that observation window, not a permanent traffic forecast.

—log lines analysedin one hour

—requests rejected—

—challenges recordedbrowser verification events

—rejections on the search route/catalogo/Search/Results

Identified agents

Two declared crawlers concentrated the rejected traffic.

The active rule classified requests by User-Agent and left a structured log of the applied decision.

GPTBot—

Amazonbot—

Operation and observability

Every decision leaves an auditable trace.

Protection is not limited to rejecting requests. Structured logs and metrics make it possible to understand which rule acted, where pressure is concentrated and how policies should evolve.

LOG

Structured decisions

Each event can record host, method, route, declared agent, forwarded address and the rule that produced ALLOW, DENY or CHALLENGE.

rule · host · path · user_agent

METRICS

Continuous measurement

The metrics endpoint can feed monitoring systems such as Prometheus to track requests, challenges, rejections, latency and service behaviour.

:9090 · Prometheus

OPS

Policy maintenance

Teams can review agents and trusted networks, test each route separately and keep policies and multihost mappings under version control.

review · test · version

LA Referencia development

One protection layer for multiple sites.

LA Referencia extended Anubis so that one instance can select different backends from the requested host. The mapping is centralised in a mounted YAML file, periodically checked for changes and applied without distributing routing logic across separate installations.

One auditable configuration for hosts and destinations.
Dynamic selection of the backend from the requested domain.
Less container duplication across the institutional infrastructure.

hosts.yaml

hosts:
  waf.lareferencia.info:
    http://backend-search:80
  cursos.lareferencia.info:
    http://backend-courses:80
  portal.lareferencia.info:
    http://backend-portal:80

↻

The mapping is checked periodically and updated when the file changes.

Auditable implementation

From deployment to a verified decision.

The public story is supported by a reproducible operational path. Open each stage to review the configuration, screenshots and evidence retained in this repository.

Deploy the shared networkRun Nginx Proxy Manager and Anubis with Docker Compose.

Both containers communicate through an internal Docker network. Proxy Manager publishes HTTPS; Anubis receives the policy as a read-only mounted file and forwards authorised traffic to the backend.

POLICY_FNAME: /data/cfg/botPolicy.yaml
TARGET: http://backend:8181
DIFFICULTY: "5"

Configure the proxy hostPreserve host, scheme and client address before forwarding.

A custom location sends the domain to Anubis and retains the headers required for routing and traceability.

Nginx Proxy Manager proxy host — Proxy host

Nginx custom location configuration — Custom location and headers

Long challenge headers

Challenge cookies and extensive user agents may require larger Nginx buffers. The validated configuration added these directives to the Proxy Manager HTTP context.

client_header_buffer_size 16k;
large_client_header_buffers 4 32k;

Order the traffic policyExceptions first, explicit rejects next, browser challenge last.

Anubis applies the first matching YAML rule. This makes ordering part of the policy and allows interoperability routes to be evaluated before generic browser traffic.

bots:
  - name: oai-server
    action: ALLOW
  - name: block-ai-scrapers
    action: DENY
  - name: generic-browser
    action: CHALLENGE

Routes to evaluate explicitly

Depending on the portal, APIs, sitemaps and static resources can also be kept outside the challenge. The final matrix should be tested and documented for each service.

/api · /sitemap.xml · /themes · /css · /js · /images

Authorise trusted networksMake the decision in Nginx before the request reaches Anubis.

Institutional networks, recognised indexers and internal monitors can be identified with Nginx geo rules and routed directly to the final service.

Trusted network configuration — Example of trusted-network configuration

Verify and observeTest HTTP 403, challenge flow, allowed routes and structured logs separately.

A controlled request declaring GPTBot produced an explicit deny. The log identifies the host, path, agent and rule, distinguishing a policy decision from a backend failure.

HTTP 403 response for a rejected crawler — HTTP 403

curl -A "GPTBot" https://lareferencia.info

"msg": "explicit deny"
"rule": "DENY"
"name": "bot/block-ai-scrapers"

Adapt the challenge experienceBuild an institutional image without changing the protected portal.

The challenge assets were rebuilt with LA Referencia branding and validated for the implementation environment.

Anubis challenge with LA Referencia branding — Institutional challenge page

Evidence and scope

What this demonstration documents.

This page synthesises the implementation validated by LA Referencia between May and July 2026. The operational evidence combines controlled requests, structured logs and one observed hour on an Internet-facing VuFind instance. Traffic patterns can change; the figures remain tied to the stated observation window.

The implementation mitigates automated scraping; it is not presented as a complete application-security layer.
The deployed core uses open-source Anubis and Nginx Proxy Manager functionality.
Policies, trusted networks and multihost destinations remain locally controlled and versionable.

Read the technical report ↗

Code and documentation

Open components.
Traceable implementation.

The demonstration connects the public explanation with the source projects, the LA Referencia multihost development and the complete technical record.

LA REFERENCIAAnubis with multihost support

Derived repository containing the extension developed for multiple domains and backends.

Open code ↗ UPSTREAMAnubis

Open-source project used for bot classification, explicit rejection and computational challenges.

Open project ↗ UPSTREAMNginx Proxy Manager

Entry point for HTTPS, certificates, domains, trusted networks and routing.

Open project ↗ REPORTComplete technical report

Architecture, configuration, validation, evidence, scope and technical continuity.

Read report ↗

Project context

LA Referencia IOI

Work developed by the LA Referencia team within the project “Expanding and innovating federated open science infrastructure in Latin America”, supported by Invest in Open Infrastructure.

Explore the project and its progress ↗