Imagine a large language model analyzing thousands of medical research papers, automatically identifying every medication mentioned, every reported side effect, every disease correlation. The goal: turn a mountain of unstructured text into structured, queryable knowledge. Drug A interacts with Drug B. Treatment X reduces symptom Y in patients with condition Z.
This is a real use case. Organizations across pharma, healthcare, and life sciences are actively pursuing it. And at first glance, the path seems obvious: plug in an LLM, write a prompt, collect the results.
What actually happens in practice is more interesting and more instructive. And the pattern we see consistently is this: the model produces output immediately. Getting that output to be correct, stable, and production-ready takes considerably longer.
This article gives an insight in why information extraction has been and still is a major challenge in computer science. We show that LLMs not simply solve all these challenges for us. We can use AI to tackle challenges in a way we couldn’t before. Still we are the once grasping the challenge fully and advising the models on strategies on how to cope with them. Still a model needs human-defined schemas, domain glossaries, and review to handle the real computer-science challenges of entity extraction reliably.
The Task Sounds Simple. It Isn’t.
Named Entity Recognition (NER), also called entity extraction, is the task of identifying structured pieces of information within unstructured text and classifying them correctly.
In the medical example above, that means recognizing metformin as a medication, systolic blood pressure as a clinical measurement, and control group as a study construct all within the same sentence, without confusing them with each other.
In a procurement context, it means extracting a deadline from a 40-page tender document, even when it appears in different formats across different portals, under different headings, surrounded by different amounts of boilerplate text.
The challenge is the same regardless of the domain. The specific difficulty varies enormously. The correctness of exraction is crucial.
If Santa used NER to evaluate and structure wishes from letters and lists, he could process them fast enough to celebrate Christmas twice a year – even right now in July. But speed alone won’t help if wishes are misunderstood (low Precision) or missed entirely (low Recall). Either failure mode would tank Santa’s performance rating (F1-Score).
Three Challenges That Don’t Go Away
These are the problems that surface in almost every entity extraction project, regardless of whether the solution uses rules, classical machine learning, or a modern LLM.
1. Ambiguity
The word “Apple” is a brand and a fruit. The word is identical in both cases. The correct interpretation depends entirely on context. One term, two meanings
In biomedicine alone, the Unified Medical Language System (UMLS) lists over 18 million concept names. Synonyms, abbreviations, brand names, and spelling variations spread across more than 3.5 million distinct medical concepts. The same term can refer to completely different diseases, and the same disease can appear under dozens of different names. Multiple terms, one meaning.
Getting these wrong doesn’t just produce a bad label. It can change the meaning of an entire sentence.
2. Nested Entities
Some entities contain other entities. “Lincoln Memorial” is a location, but it also contains Lincoln, which could be classified as a person. Depending on what you’re extracting and why, both readings are valid.
A system that only looks for flat, non-overlapping entities will systematically miss the nuance here. Handling nested entities requires the model to reason at multiple levels simultaneously.
3. Discontinuous Entities
“The patient experienced muscle pain and fatigue.”
If you’re extracting symptoms, both “muscle pain” and “muscle fatigue” are entities, but they’re grammatically separated. If you need to capture them as a compound concept, their “edges” don’t align neatly. The entity is split across the sentence.
This is common in medical and legal texts. It’s one of the reasons simple keyword matching or basic NER approaches fail in specialized domains.
A Brief History: Same Problem, Different Tools
One of the most important things to understand about entity extraction is that it isn’t a new problem. It has been an active area of research for decades and the history of how the field has approached it explains a lot about where LLMs fit in today.
Rule-based systems came first. Define a list of known entities, write patterns for what surrounds them, and match against text. Effective for narrow, stable domains. Brittle the moment the vocabulary or structure changes. Expensive to maintain as complexity grows.
Machine learning approaches followed: Hidden Markov Models, Conditional Random Fields, Support Vector Machines. Instead of writing rules by hand, these systems learned patterns from labeled examples. They generalized better than rules but they required large volumes of annotated training data, and a model trained on news articles performed poorly on medical reports because the vocabulary and entity types were completely different.
Deep learning and Transformers changed the economics around 2015–2018. Models like BERT could be pre-trained on vast general corpora and then fine-tuned on domain-specific data with far fewer labeled examples. Domain-specific variants like BioBERT, SciBERT, astroBERT emerged for medicine, science, and astronomy respectively. These represented genuine state-of-the-art performance for their target domains.
LLMs and prompt engineering are the current chapter. Well-known models can tackle extraction tasks with nothing more than a well-crafted prompt – no fine-tuning, no labeled dataset, no GPU cluster. This is a meaningful reduction in the cost of entry.
But here is the thing worth saying clearly: at each transition, the underlying challenges didn’t disappear. They were addressed differently. Ambiguity, domain dependency, nested entities were problems in 2003 and they are problems in 2026. The tools evolved. The science didn’t vanish.
Where LLMs Actually Struggle: Two Scenarios
The most useful framing we’ve found is distinguishing between simple NER and complex NER, because they are genuinely different problems that require different solutions.
Simple NER: Predictable Entities in Stable Formats
A medical bill has predictable structure: provider name, date of service, procedure codes, amounts. The entities are well-defined, consistently formatted, and don’t require reasoning beyond the text in front of you.
For this kind of task, a fine-tuned smaller model might outperforms a general-purpose LLM. The investment in domain-specific fine-tuning might pay off because the entity types don’t change often and the volume justifies it.
Complex NER: When Extraction Requires Reasoning
Now consider a building specification document. The kind issued as part of a public construction contract. The task: extract how many smoke detectors are delivered.
The text might read:
“One smoke detector per room.”
To answer the question, the model needs to know how many rooms there are. That information might be in an attached floor plan. It might need to be inferred from other sections. The entity isn’t a name or a date. It’s the result of a multi-step reasoning process that spans multiple pieces of context.
Or the document might say:
“2 on the ground floor and 3 in the attic.”
Now the model needs to recognize that these refer to the same entity type and sum them. Five smoke detectors total.
This is no longer pattern matching. This is reasoning. And that distinction determines the entire architecture of a viable solution.
The Architecture That Actually Works for Us
Recognizing this distinction leads directly to a more honest and more effective design. Reminder: there is no universal best way to solve this. We are simply representing our findings and what worked best for us so far. It remains vital to keep up with ongoing research in the field of NER. When we build production-ready extraction systems for complex domains, we typically separate three concerns:
LLMs for Semantic Extraction
The LLM’s role is semantic interpretation: reading a chunk of text and identifying what it means, not just what it says. This is where modern models genuinely excel. Understanding context, resolving ambiguity within a passage, recognizing paraphrase and synonymy.
The critical insight from both our project experience and from published research: prompt design matters more than most teams expect. A naive prompt (“extract all entities”) underperforms significantly. A well-constructed prompt with explicit entity definitions, domain-specific synonyms, and worked examples can close much of the gap with fine-tuned models without the training cost and knowledge.
For developers: This is the prompt engineering layer. Think zero-shot baseline → few-shot with curated examples → self-verification step. Each iteration should be measured against a held-out evaluation set, not eyeballed.
Vector Databases for Contextualization
For entities that require external knowledge (e.g. room counts from a floor plan or drug interaction data from a reference database) the model needs access to relevant context at query time.
Retrieval-Augmented Generation (RAG) addresses this: a query is matched against a pre-processed knowledge base, the most relevant chunks are retrieved, and the LLM receives both the extraction task and the supporting context in the same prompt.
This is how you bridge the gap between what’s written in the document and what needs to be known to answer the question correctly.
For developers: Embedding model choice and chunk size have a large impact on retrieval quality. Evaluate retrieval separately from generation. A failing RAG pipeline usually breaks at the retrieval step, not the generation step.
Pipeline Orchestration for Reliability
The full system – document ingestion, chunking, embedding, retrieval, LLM inference, output validation, storage – is a pipeline, not a single API call. Managing this as a pipeline rather than a monolith makes each step testable, observable, and replaceable independently.
AI workflow automation platforms make it practical to wire these steps together in a way that operations teams can actually maintain with logging, error handling, retry logic, and clear handoff points between components.
Our Takeaway
AI is like a calculator: just because the tool produces results doesn’t mean the underlying problem has been solved. The challenges that have defined entity extraction for decades – ambiguity, domain dependency, nested and discontinuous entities, the need for external knowledge are still present. What changes is how we address them, and how accessible the tools for doing so have become.
We see this as one of the most interesting spaces in applied AI right now: the gap between “the LLM gives me something” and “the system reliably gives me the right thing” is where the real engineering happens and where the real business value is created.
If you want to read more about this topic:
- Is ChatGPT a General-Purpose NLP Task Solver? — https://arxiv.org/pdf/2302.06476
- FsPONER: Few-shot Prompt Optimization for NER in Domain-specific Scenarios — https://arxiv.org/html/2407.08035v1
- Leveraging Large Language Models for Rare Disease Named Entity Recognition — https://arxiv.org/pdf/2508.09323
- Advancing entity recognition in biomedicine via instruction tuning of LLMs (Oxford Bioinformatics) — https://academic.oup.com/bioinformatics/article/40/4/btae163/7633405
- Optimized BERT-based NLP outperforms Zero-Shot Methods for Automated Symptom Detection in Clinical Practice — https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12689901/
- Supervised Fine-Tuning or In-Context Learning? Evaluating LLMs for Clinical NER — https://arxiv.org/pdf/2510.22285
- Comparative Study of Pre-Trained BERT and LLMs for Code-Mixed NER — https://arxiv.org/pdf/2509.02514
- Local LLM Ensembles for Zero-shot Portuguese NER — https://arxiv.org/html/2512.10043
- Bridging Zero-Shot and Fine-Tuned Performance in Text Classification (ASRJETS) — https://asrjetsjournal.org/American_Scientific_Journal/article/download/12048/2893/28507
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm — https://arxiv.org/pdf/2102.07350
- The Few-shot Dilemma: Over-prompting Large Language Models — https://arxiv.org/html/2509.13196v1
- Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shot — https://arxiv.org/html/2506.14641v3
- Do LLMs Overcome Shortcut Learning? — https://arxiv.org/pdf/2410.13343
- Self-Generated In-Context Learning (Learn Prompting) — https://learnprompting.org/docs/advanced/few_shot/self_generated_icl
- Meta In-Context Learning Makes LLMs Better Zero and Few-Shot Relation Extractors — https://arxiv.org/pdf/2404.17807
- An Explanation of In-context Learning as Implicit Bayesian Inference — https://arxiv.org/pdf/2111.02080
- Large Language Models Are Human-Level Prompt Engineers — https://arxiv.org/pdf/2211.01910
- Language Models are Few-Shot Learners (GPT-3 paper) — https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
- Nested Named Entity Recognition: A Survey (ACM TKDD) — https://dl.acm.org/doi/10.1145/3522593
- Nested Named Entity Recognition: A Survey of Latest Research (Expert Systems, 2025) — https://onlinelibrary.wiley.com/doi/abs/10.1111/exsy.70052
- Nested Entity Recognition Method Based on Multidimensional Features and Fuzzy Localization — https://link.springer.com/article/10.1007/s11063-024-11657-2
- PNER: Applying the Pipeline Method to Resolve Nested Issues in NER — https://www.mdpi.com/2076-3417/14/5/1717
- GapDNER: A Gap-Aware Grid Tagging Model for Discontinuous NER — https://arxiv.org/pdf/2510.10927
- On Fusing ChatGPT and Ensemble Learning in Discontinuous NER in Health Corpora — https://arxiv.org/pdf/2412.16976
- Linguistically Informed Relation Extraction and Neural Architectures for Nested NER (BioNLP-OST 2019) — https://arxiv.org/pdf/1910.03385
- What is Named Entity Recognition? (Activeloop glossary overview) — https://www.activeloop.ai/resources/glossary/named-entity-recognition-ner/
- UMLS 2025AB Release Available (NLM) — https://www.nlm.nih.gov/pubs/techbull/nd25/nd25_umls_release_ab_beta.html
- MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts — https://arxiv.org/pdf/1902.09476
- Integrating UMLS Knowledge into Large Language Models for Medical Question Answering — https://arxiv.org/pdf/2310.02778
- SECNLP: A Survey of Embeddings in Clinical Natural Language Processing — https://arxiv.org/pdf/1903.01039
- Unified Medical Language System overview (ScienceDirect Topics) — https://www.sciencedirect.com/topics/biochemistry-genetics-and-molecular-biology/unified-medical-language-system
- Description and Evaluation of Semantic Similarity Measures Approaches — https://arxiv.org/pdf/1310.8059
- Exploiting the UMLS Metathesaurus for extracting signs/symptoms (ScienceDirect) — https://www.sciencedirect.com/science/article/pii/S1532046415001926
- Using UMLS metathesaurus concepts to describe medical images: dermatology vocabulary — https://www.sciencedirect.com/science/article/abs/pii/S0010482504001155