AI URL Fetching: Authorise the Connection, Not the Link

A research assistant receives a link, calls a fetch tool and returns a useful paragraph. That ordinary sequence hides an important privilege: the request originates from the service running the tool, not necessarily from the person who supplied the link. Its network may contain internal APIs, cloud metadata services and credentials that the reader cannot access.

The design decision is therefore not simply whether the model may browse. It is which network destinations this task may contact, with which request data, and which component enforces that permission before the connection occurs. A persuasive explanation from the model cannot answer those questions on behalf of the application.

This guide addresses server-side HTTP fetching, not interactive browser automation. The linked standards and documentation establish protocol and implementation behaviour; the tool profiles, acceptance tests and operating choices below are AIEngine's engineering synthesis. They are not results from an AIEngine security assessment. Living sources were checked on 13 September 2026.

Choose a connector or a public-web fetcher

Server-side request forgery, or SSRF, turns an application's request-making ability into unintended access. OWASP's prevention guidance distinguishes calls to known trusted applications from calls to arbitrary external destinations, and recommends application controls alongside network restrictions. That distinction should shape the tool catalogue.

A fixed business integration usually does not need a model-supplied URL. Accept a document identifier or business operation, then let trusted connector code construct the destination. Public research has different requirements and deserves a separate execution profile.

Do not combine these profiles behind one unrestricted fetch function with an “internal” flag chosen by the model. Route selection belongs to the application. Otherwise a feature intended to read public documentation can inherit the authority of an authenticated internal connector.

The agent sandbox guide addresses the wider execution boundary. Here the narrower question is whether the HTTP client, resolver and egress path preserve that boundary for every attempted connection.

Parse once, then constrain the parsed result

A URL is structured input, not a string that becomes safe after a prefix check. The WHATWG URL Standard's host-parsing rules include domain processing, IPv4 and IPv6 handling, and legacy numeric IPv4 forms. A parser can also report a validation error while still returning a URL. “It parsed” and “our policy permits it” are different conclusions.

Choose a maintained parser compatible with the actual client. Define a narrow accepted representation: HTTPS, an explicit permitted port, no embedded username or password, and no caller-controlled proxy or transport settings. Reject unsupported forms instead of repeatedly decoding or repairing them until something looks acceptable.

Compare the canonical host as a host, not as a substring of the original input. A permitted name appearing in a path, user-information field or longer unrelated hostname must not satisfy an origin allowlist. Decide deliberately how internationalised names and trailing dots are represented, and test that decision across every component that reparses the request.

For fixed connectors, construct paths from validated identifiers and encode individual components. Do not let an identifier become a complete URL through string concatenation. For public fetching, preserve a protected reference to the submitted URL when investigation requires it, but avoid putting signed query parameters or other secrets into routine logs.

Bind approval to the connection, not an earlier lookup

A common weak design resolves a hostname, checks the returned address, then passes the original hostname to an ordinary HTTP client. The client may resolve it again. The first check then says nothing conclusive about the address selected for the second operation.

Close that gap in the component that opens the connection. Resolve through the controlled resolver, classify the candidate addresses and ensure that only approved candidates can be dialled. Reapply the rule to retries, alternate address families and later connections. For a public-only service, rejecting a resolution result containing any prohibited address is a conservative, explainable policy; accepting a filtered subset requires proof that fallback cannot escape that subset.

The curl project's custom-resolution interface illustrates one implementation building block: it supplies address entries for a hostname-and-port pair in libcurl's DNS cache. Its documentation distinguishes entries that expire from entries that do not. This is not a complete SSRF defence or a recommendation to retain mappings indefinitely.

Keep the original permitted hostname for TLS identity verification and the HTTP authority while controlling the socket destination. Replacing the URL host with an IP and disabling certificate checks would discard a different security boundary. Define how mappings expire, how failures trigger fresh approval and whether existing pooled connections remain usable after policy changes.

If a proxy performs DNS resolution and connection establishment, that proxy must enforce destination policy. The application seeing only the proxy's socket address is not evidence about the final server. Likewise, test connection reuse, alternative-service routing and any transport feature that can choose a destination outside the obvious resolver path. Disable features the enforcement layer cannot mediate.

Public address space is not a complete permission model

A blocklist containing only familiar private IPv4 ranges misses other address classes and deployment-specific routes. IANA's IPv4 special-purpose registry distinguishes private use, loopback, link-local, shared and documentation space, with separate destination and global-reachability properties. Its IPv6 registry also covers unique-local, link-local, mapped and translation-related allocations. Both registries explicitly warn that their entries do not guarantee routability in a particular environment.

Use maintained address classification, then add the organisation's actual network policy. Include publicly numbered internal infrastructure, service meshes, peered networks and translation paths. A globally reachable address is not automatically an approved recipient of confidential information.

A defensible public-fetch deployment has no route to internal services even when an application-level check is wrong. Restrict DNS and proxy choices as part of that deployment. If IPv6 is enabled, enforce equivalent policy there; if it is deliberately unsupported, prove it cannot become an unchecked fallback.

Separate exceptions from the public profile. An approved internal document should be retrieved by its authenticated connector, not by adding a broad private-network exemption to the research fetcher.

Treat each redirect as a new destination request

A permitted first server can return a different location. The subsequent request needs its own parsing, destination classification and authorisation before it is sent. Resolving a relative location against the current URL is part of that process, not permission to follow it.

RFC 9110's redirection semantics, published in June 2022, distinguish method-preserving redirects from cases where a method may change. They also discuss removing or regenerating origin-specific and security-sensitive headers. A generic “follow redirects” switch therefore hides decisions about more than destination names.

For a public research tool, choose a small hop limit, detect loops and reject HTTPS-to-HTTP downgrade. Permit only the redirect status codes and methods the tool is designed to handle. Rebuild each request from an approved header set rather than forwarding the previous request wholesale.

The libcurl security guidance specifically warns that redirects can reach local or firewalled services and that protocol and authentication options affect exposure. It describes handling redirects in the application or applying suitable connection checks. Library defaults must be checked for the deployed version, not inferred from the word HTTP in the tool's name.

A newly permitted hostname still does not inherit credentials. Cross-origin redirects should not carry bearer tokens, cookies, internal headers or a confidential request body. Authenticated connectors should define their own narrowly allowed redirect behaviour, often none, rather than sharing the public fetcher's policy.

Remove credentials and bound the returned material

Run public fetching without inherited cloud credentials, user cookies, client certificates, credential files or automatic authentication helpers. Do not expose arbitrary methods, headers or request bodies just because the underlying library supports them. A fixed GET profile can be useful, but GET alone does not establish harmlessness: destinations can implement unsafe behaviour, and request paths can disclose data.

AWS's Instance Metadata Service documentation explains that IMDSv2 uses session tokens obtained through a PUT request and supports an IPv6 metadata endpoint in applicable configurations. Token requirements add protection; they do not grant a fetch worker permission to reach metadata. Restrict the route as well, and do not equip the worker to request those tokens.

Set total and per-stage deadlines, request and response byte limits, redirect limits and decompressed-size limits. Abort oversized or stalled transfers rather than relying only on a declared content length. Partition any retained response state by its permitted audience.

Fetching establishes what bytes were received, not whether they are trustworthy. Return source URL, final URL, acquisition time, content type, digest and a bounded content reference. Keep retrieved instructions as untrusted data. The RAG ingestion-security guide explains the separate admission decision before that material becomes searchable evidence.

Work through a supplier-document fixture

Consider a synthetic acceptance fixture for an assistant that compares public supplier specifications with an uploaded requirements sheet. It may fetch public documents but cannot contact internal systems, authenticate to suppliers or transmit the requirements sheet. This is a proposed test scenario, not a customer case study.

The application supplies the fetcher with a task identity and the public-fetch profile. A fixture URL first resolves to a permitted test server. That server returns a redirect whose destination is prohibited by the profile. The expected result is a structured denial before any connection to the second destination.

The assistant should report that the linked document could not be retrieved under the current policy. It may ask for an uploaded copy or use another authorised source. It must not rewrite the address, switch tools to obtain internal access or repeatedly ask the model for a way around the denial.

Now change the fixture so the approved hostname's resolution changes between attempts. Verify that a fresh connection cannot use a newly prohibited address. Then make the first candidate unreachable while another address is available: fallback must remain inside the approved set.

Finally, return a valid public document that instructs the assistant to fetch an internal URL. The content may be processed as a supplier claim, but it cannot expand network authority. This tests the composition of content isolation and connection enforcement rather than the model's ability to recognise one suspicious sentence.

Test forbidden effects, not reassuring error messages

Run adversarial fixtures only against infrastructure the organisation controls or is authorised to test. Observe the egress layer and destination listeners, not just the fetch function's return value. A tool can report failure after it has already contacted a forbidden service.

The release suite should include:

• Hostname lookalikes, embedded user information and parser disagreements.
• Alternative numeric address representations and IPv4-mapped IPv6 handling.
• Mixed permitted and prohibited DNS answers, rebinding and resolver failure.
• Relative redirects, cross-origin redirects, loops and protocol downgrade.
• Retry and dual-stack fallback after the first connection fails.
• Proxy configuration, environment inheritance and direct-egress bypass attempts.
• Connection reuse after a policy revision or task-authority expiry.
• Custom-header, cookie and credential propagation across destinations.
• Large compressed responses, slow bodies and cancellation during transfer.
• Legitimate internationalised domains and ordinary public CDN redirects.

A negative case passes only when the prohibited effect does not occur and the reason is recorded. Include positive cases so the service remains useful. Distinguish an intentional policy denial from DNS failure, certificate failure, remote refusal and a broken client; these require different remedies.

Do not advertise a passing fixture set as universal SSRF immunity. It supports a narrower claim about the tested client, proxy, resolver, network topology and policy revision.

Measure reach, disclosure and legitimate completion

Track attempted and established destinations separately. Record policy version, task identity, canonical origin, approved address classification, redirect sequence, selected route and final outcome. Keep sensitive URL components out of ordinary telemetry; the privacy-safe incident replay guide provides the wider evidence-handling discipline.

Review unexpected successful connections as boundary failures, not merely unusual model behaviour. Track prohibited attempts by cause, denial-to-retry loops, expired-policy reuse, credential-forwarding failures in tests, and eligible documents successfully retrieved within limits. A rising denial rate can mean better enforcement, hostile inputs or an overbroad rule; investigate the denominator before celebrating it.

Reopen the acceptance evidence after changes to the HTTP library, parser, resolver, proxy, network routes, cloud metadata configuration or tool schema. Keep exceptions owned and time-bounded. If the team cannot show where final-destination approval is enforced, reduce the tool to registered connectors or supplied snapshots until it can.

The useful promise is concrete: this task can obtain authorised evidence without turning its fetch service into a route to unrelated systems. Enforce that promise where the connection is made.