How to Use Ollama Locally for Private AI Writing

You're halfway through a sensitive draft when the useful sentence appears. It might be a contract clause, a private journal entry, a product idea, or a comment inside proprietary code. Then you remember that a cloud writing assistant may process the text on someone else's infrastructure, and the convenience suddenly feels less attractive.

Ollama gives you another path. It runs supported language models on your own machine, while tools such as RewriteBar can send selected text to that local model for grammar fixes, rewrites, translations, and custom workflows. You still need to respect hardware limits and model quality, but the workflow can keep private writing offline and available without an internet connection.

Why Run AI Models on Your Own Machine

Cloud AI is convenient because the provider handles model hosting, memory, acceleration, updates, and availability. Local AI reverses that arrangement. You install a model runner, download a model, and let your computer handle inference. In exchange, you gain more control over where your text goes and when the system remains available.

For sensitive writing, that control is the main reason to learn how to use Ollama locally. A draft sent to a local Ollama endpoint stays within your own environment, rather than being submitted to a remote provider as part of a cloud request. That doesn't make every local setup automatically secure, since other applications or users on the machine may still have access, but it removes the cloud transfer from the normal workflow.

Offline access matters just as much. A local model can assist with editing on a flight, in a restricted workspace, or during an outage, provided the model has already been downloaded and the machine can run it. Local inference also avoids per-token billing from a cloud API, although electricity, storage, hardware, and your time still have real costs.

Practical rule: Treat local AI as a private, self-managed service, not as a free version of a cloud model.

What Ollama actually manages

Ollama provides the local runtime and model-management layer. Its model library packages models in variants with published parameter sizes, so you can choose a model that matches your machine instead of trying to fit one fixed cloud-scale system into every device. The library includes Llama 3.1 variants at 8B, 70B, and 405B parameters, illustrating the range between responsive local models and much larger hardware-intensive options.

That choice is important because model size affects memory pressure, loading behavior, and responsiveness. A smaller model may be practical for quick grammar corrections, while a larger model may produce stronger reasoning or more nuanced rewrites if your hardware can sustain it. Ollama's catalog was updated as recently as August 19, 2026, according to the library page, so the ecosystem is actively maintained rather than frozen around a single collection.

Set expectations before installing

Check available memory, free disk space, and whether your machine has usable GPU acceleration. You don't need an enthusiast workstation to begin, but you do need enough headroom for the model and the operating system to work together. If a model barely fits, the result may be technically functional but frustratingly slow.

For a practical starting point, choose a smaller model first, test it with your real writing, and only move upward when you know what quality or context length you need. A local writing assistant succeeds when it responds reliably inside your existing workflow, not when it runs the largest model you can download.

If you want a broader explanation of privacy-focused model use before installing anything, read this guide to offline AI models.

Installing Ollama and Pulling Your First Model

On macOS, the simplest route is to download Ollama from its official website, open the installer, and allow the application to run. Ollama provides a command-line interface and a background service, so you can use the terminal for testing while other applications connect through its local API.

A MacBook Pro on a wooden desk displaying the Ollama installation command in a terminal window.

After installation, open Terminal and check that the command is available:

ollama --version

Launch the Ollama application if it isn't already running. Then pull a starter model from the library. The exact model you choose depends on your hardware, but the workflow looks like this:

ollama pull llama3.1:8b

The pull command downloads the model files to your computer. That download can take time and consume substantial storage, particularly when you keep several model variants. The Ollama model library shows published sizes and available tags, so check the selected model before beginning and leave room for the operating system, applications, and additional models.

Run the model interactively:

ollama run llama3.1:8b

You should see a prompt. Try something close to your daily work:

Rewrite this paragraph for clarity while preserving its meaning.

The first response may feel slower because Ollama has to load the model. Later prompts can behave differently once the model remains available in memory. You can inspect downloaded models with:

ollama list

And you can see active models with:

ollama ps

The local server normally listens on localhost:11434, which is the address that compatible applications use. For a writing tool, you generally won't need to issue terminal prompts after setup. You'll select the local provider, choose the downloaded model, and let the application send requests to Ollama.

For a visual walkthrough of the setup process, use the following video after you've reviewed the basic commands:

If you're connecting a writing assistant rather than using the terminal, the Ollama provider setup documentation explains the application-side configuration.

Choosing the Right Model Size for Your Hardware

Model selection is where many local setups go wrong. Beginners often download an impressive-looking model, discover that it technically starts, and then mistake poor responsiveness for a broken installation. The better approach is to match the model tier to available memory, not just the computer's advertised specifications.

Independent hardware guidance places small models around 8 GB of RAM, makes 7B and 13B models more practical with 16 GB or more, and often puts 30B-plus models at 32 GB or more of system memory or on a high-VRAM GPU. These are planning rules rather than guarantees, because quantization, context length, operating-system load, and GPU offload all affect the result. The Ollama hardware guide also warns that model files can consume tens of gigabytes and that performance can fall sharply when the model doesn't fit cleanly into available RAM or VRAM.

Model Size to Hardware Mapping

Available Memory	Recommended Model Size	Example Models	Expected Experience
About 8 GB RAM	Small models	`gemma3:4b` or another compact variant	Suitable for short edits and lightweight prompts, with limited room for long context
16 GB or more	7B to 13B class	`llama3.1:8b`	A practical starting tier for writing, coding help, and general assistance
32 GB or more, or a high-VRAM GPU	30B-plus models	Larger library variants	Higher capability may be available, but loading and response speed depend heavily on hardware
64 GB or more on a high-end Mac	Larger models	`gemma3:27b`	More ambitious local use is possible when memory and storage have sufficient headroom

The final row reflects guidance from Northwestern's local LLM guide, which distinguishes high-end Macs with 64GB or more of RAM from most laptops and desktops. Its practical recommendation is to use a larger model such as gemma3:27b on the former and a smaller model such as gemma3:4b on the latter.

A model that answers quickly and consistently is more useful than a larger model that forces you to wait between every sentence.

Quality versus responsiveness

For writing assistance, a compact model can be enough for grammar, spelling, tone adjustments, and short rewrites. More involved tasks, such as preserving a complex argument across a long document or following a detailed multi-step workflow, put greater pressure on memory and context handling.

Start with a model in the tier your computer can run comfortably. Test it inside your real writing workflow before downloading a larger alternative. If the model struggles with a task, decide whether the problem is model capability, prompt design, context length, or hardware saturation. Increasing model size isn't always the correct fix.

Connecting Ollama to RewriteBar and Local Apps

A terminal confirms that Ollama runs. A connected writing tool determines whether it helps during the workday. Ollama provides an OpenAI-compatible chat API, so existing applications can often use a familiar integration pattern instead of requiring a separate local connector. Its milestone history records OpenAI-compatible chat API support on February 8, 2024, a native Windows preview on February 15, 2024, AMD GPU preview support on March 14, 2024, and structured outputs through JSON Schema in November 2024. These developments help local models fit productivity and developer workflows, as documented in Ollama's milestone history.

For RewriteBar on macOS, start Ollama and confirm that the required model is downloaded. In RewriteBar's preferences, choose the Ollama provider, enter the local OpenAI-compatible base URL, and select the exact model tag returned by ollama list. The endpoint commonly used for this integration is http://localhost:11434.

Screenshot from https://rewritebar.com

A practical writing loop

With the provider configured, highlight text in any application where you write and trigger RewriteBar with its keyboard shortcut. Send a paragraph for grammar correction, request a clearer tone, or run a custom workflow that preserves technical terminology. The selected text is processed by your local Ollama service instead of a remote writing API.

This setup supports several practical workflows:

Sensitive editing: Polish contract language, private notes, or unpublished product material without sending the selection to a cloud provider.
Offline translation: Use downloaded models without an internet connection, while allowing for differences in language coverage and translation quality.
Developer writing: Rewrite code comments, explain an error message, or turn rough implementation notes into a clearer issue description.
Repeatable transformations: Apply structured prompts for summaries, user stories, or formatting cleanup without rewriting the same instruction each time.

Local processing also has limits. A smaller local model may change the meaning of a rewrite or miss an instruction that a frontier cloud model handles reliably. Shorten the selection, make the request more explicit, or test another downloaded model before assuming the integration is broken. The guide to using RewriteBar with Ollama explains the provider settings and application workflow in more detail.

Other local applications follow the same pattern. Point an OpenAI-compatible client to the Ollama base URL, use the exact model name installed on the machine, and test a short prompt before building a larger workflow. This keeps private writing assistance available inside the tools where the work already happens.

Benchmarking Your Local Setup for Real Performance

“Fast enough” depends on the task. A short grammar correction may feel responsive even on modest hardware, while a long technical document can expose slow prompt processing before generation begins. Measure the workflow you intend to use, rather than relying on a model name or a vague impression from one terminal response.

Run a model with verbose output and record the values Ollama reports. The useful fields are:

eval rate: Generation throughput, expressed as generated tokens per second.
prompt eval rate: Prefill speed, which indicates how quickly the model processes the input prompt.
load duration: Time spent loading the model before generation begins.

The local AI benchmarking guide recommends using a real prompt and comparing cold and warm runs. A cold run includes model-loading overhead, while a warm run gives you a clearer view of steady-state generation. Keep both measurements because users experience both, especially when the model is unloaded between tasks.

Test the bottleneck that matters

Use a short prompt, then repeat the test with the longest input your writing workflow normally sends. A model can generate quickly after loading but still feel slow when it has to prefill a long context. That distinction matters for RewriteBar workflows involving full paragraphs, pasted documents, or multi-step instructions.

Keep the test conditions consistent. Use the same model, prompt, context length, application load, and output expectation when comparing runs. Record the results in a small note so you can tell whether a model change improved actual usability or merely changed the first-response delay.

A representative benchmark cited in independent comparisons found Ollama reaching roughly 62 tokens per second with Llama 3.1 8B on an RTX 4090 in a single-stream test, close to direct llama.cpp and below vLLM, as reported in the Ollama performance comparison. That result is useful as a reference point, not a promise for your computer. Your processor, GPU, memory placement, quantization, prompt length, and background applications can all change the outcome.

Networking and Remote Access Configuration

Ollama works with the fewest moving parts when the client and model share one machine. Its default local binding keeps the service reachable only from that computer. The setup changes when a Docker container, another laptop, or a separate writing application must send requests to Ollama.

The official Ollama FAQ documents OLLAMA_HOST, which controls the address Ollama binds to. For a temporary Linux or macOS test, stop and restart Ollama with:

export OLLAMA_HOST=0.0.0.0
ollama serve

Bind to a specific private address instead of every interface when your environment supports it. Restrict the firewall to trusted devices or a protected network, because an exposed Ollama service may not provide the authentication you expect from a hosted API.

The Docker localhost trap

Inside a container, localhost means the container itself. A request to http://localhost:11434 can therefore fail even though Ollama responds normally on the host.

Configure the client to use http://host.docker.internal:11434, where that gateway name is supported, or use a host address reachable from the container. Test the route before changing the model or prompt:

curl http://host.docker.internal:11434/api/tags

A successful response lists locally available models. If curl returns Could not resolve host, the container cannot resolve the gateway name. If it returns Connection refused, Ollama is probably still bound to localhost, is not running, or the selected port is blocked. Check the host service first, then confirm the value of OLLAMA_HOST and restart Ollama.

For RewriteBar and Ollama on one Mac, keep the service local. Remote access should be deliberate, protected by firewall rules or a VPN, and limited to the devices that need private offline writing assistance. A quick network workaround can expose every model endpoint on that interface.

When Local AI Beats Cloud and When It Does Not

Local Ollama is a strong fit when privacy, offline access, and predictable usage matter more than maximum model capability. It works particularly well for recurring transformations such as grammar correction, tone changes, code comments, short translations, and drafts that shouldn't leave your device. Once the model is downloaded, those tasks don't require a per-token cloud request.

Cloud AI still wins in several situations. Hosted providers can offer larger models, broader managed context, faster infrastructure, and capabilities that won't fit comfortably into consumer hardware. They also remove the work of downloading models, managing memory pressure, updating runtimes, and diagnosing local performance.

Use a local model when the text is sensitive, the task is repeatable, the computer can run the chosen model comfortably, and a little setup is worth the control. Use a cloud model when the task needs advanced reasoning, very long context, current external information, or capacity your machine can't provide. A mixed workflow is often the most practical: local Ollama handles private everyday writing, while a cloud provider handles occasional heavyweight jobs after you've considered the data involved.

The best local setup isn't the one with the largest model. It's the one that responds reliably inside the tools where you already write.

RewriteBar brings local Ollama into a macOS menu-bar writing workflow, so you can fix grammar, adjust tone, translate text, and run custom actions without leaving the app you're using. Visit RewriteBar to connect a private local model and make offline writing assistance part of your daily routine.