AI Glossary
The complete dictionary of Artificial Intelligence
Multi-annotator Active Learning
Machine learning paradigm where multiple annotators collaborate to label the most informative data selected by an active algorithm. This approach optimizes the use of human resources by leveraging complementary expertise and variable annotator costs.
Collaborative Sampling Strategy
Sample selection method that coordinates the preferences and expertise of multiple annotators to maximize the overall quality of labels. It balances between model uncertainty and optimal task distribution according to each annotator's skills.
Annotator Expertise Model
Probabilistic or deterministic framework that quantifies the competence, reliability, and specific biases of each annotator on different classes or data types. This model guides sample allocation based on the individual strengths of each contributor.
Dynamic Sample Allocation
Adaptive process that assigns unlabeled samples to the most appropriate annotators in real-time, based on their current expertise and availability. This allocation optimizes the trade-off between annotation quality and temporal or monetary budget.
Inter-annotator Confidence Matrix
Data structure measuring the relative consistency and reliability between annotations from different contributors on similar samples. It enables the detection of systematic disagreements and adjusts confidence weights for final label aggregation.
Adaptive Annotation Cost
Variable cost model that adjusts compensation or time allocated to each annotation according to sample difficulty and required expertise. This approach optimizes budget allocation by favoring the most cost-effective assignments in terms of information gain.
Annotation Diversity
Selection criterion ensuring varied distribution of samples among annotators to avoid overspecialization and maximize feature space coverage. It maintains balance between exploiting expertise and exploring new perspectives.
Multi-source Label Aggregation
Fusion technique combining labels from multiple annotators into a single consensus prediction, using methods such as weighted voting, Dawid-Skene models, or Bayesian inference. This aggregation corrects individual errors and produces more reliable labels.
Weighted Uncertainty Sampling
Strategy that evaluates model uncertainty while weighting this measure by the cost and expertise required to resolve each sample. It prioritizes instances offering the best information/benefit ratio according to available annotator capabilities.
Multi-annotator Query-by-Committee
Extension of Query-by-Committee where multiple models are trained on different annotation subsets to identify samples with the greatest predictive disagreement. This approach is enriched by the diversity of annotator perspectives within the committee.
Annotator Calibration
Systematic adjustment process of annotator predictions to correct individual biases and harmonize confidence scales between different contributors. Calibration ensures that reliability scores are comparable and usable for dynamic allocation.
Annotation Quality Assessment
Continuous metric framework that measures the precision, consistency, and informational value of annotations provided by each contributor. These metrics feed expertise models and guide future allocation decisions.
Intelligent Sample Routing
Automated decision system that routes each unlabeled sample to the optimal annotator or group of annotators based on performance predictors and resource constraints. This routing maximizes the efficiency of the collaborative annotation process.
Semi-supervised Multi-expert Learning
Hybrid approach combining multi-annotator active learning with semi-supervised techniques to exploit large quantities of unlabeled data. It uses high-quality annotations as seeds to propagate labels throughout the dataset.
Annotator Reliability Model
Mathematical representation of the probability that an annotator produces a correct annotation, often modeled as a conditional distribution dependent on sample type. This model evolves dynamically with the accumulation of historical annotations.
Exploration-Exploitation Strategy
Decision framework balancing the assignment of familiar samples to expert annotators (exploitation) with the discovery of capabilities on new data types (exploration). This strategy avoids over-specialization while maximizing short-term quality.
Expert Opinion Fusion
Sophisticated aggregation method that combines judgments from multiple annotators by accounting for their correlation, domain-specific expertise, and performance history. It often employs Bayesian techniques or belief graphs for optimal fusion.