Large language models (LLMs) demonstrate a notable bias towards Japanese culture when responding to open-ended prompts about societal norms and traditions, according to a study by researchers from the University of the Basque Country and Cardiff University. The findings, detailed in a white paper, indicate that frontier LLMs like Claude, Gemini, and DeepSeek exhibit a "disproportionate prominence of Japan" in their outputs.
The research team designed a multilingual set of 31,680 questions across 66 cultural subtopics to assess regional and cultural biases within LLMs. These questions, which did not directly reference specific countries, probed areas such as beliefs, societal roles, education, traditional practices, and daily life. Participants were asked to provide examples from a chosen country or region.
The study observed that LLMs generally tend to respond with information pertaining to the region associated with the language of the prompt. However, when models provided references outside of the prompt's language-associated region—an exogenous reference—Japan emerged as the most frequently cited culture. On average, six out of eight models evaluated favored referencing Japan in these instances, with the United States following in second place, then India, China, and France.
Across all 24 languages tested, LLMs overwhelmingly preferred referencing Japan in seven out of eleven cultural prompt topics when excluding responses that referred to the prompt's own associated country. This pattern suggests an uneven representation of cultures in LLM outputs, with a strong concentration on a limited number of dominant regions.
Researchers also investigated how cultural bias develops during the LLM training process by comparing model responses before and after instruction tuning, a process that refines models for more helpful user interactions. Base models, prior to instruction tuning, exhibited broader cultural associations, including substantial references to Japan, India, China, and several European countries alongside the United States. However, after instruction tuning, LLMs showed a more pronounced preference, with sharp increases in alignment with the United States and Japan, and a reduction in references to most other nations. This convergence toward culturally dominant regions was observed even in models developed outside Western contexts, suggesting that post-training processes can lead to a homogenization of cultural perspectives rather than simply reflecting the model's origin.
The bias is particularly pronounced in models that have undergone supervised fine-tuning (SFT), where models are trained on curated datasets of correct responses. The study found that SFT significantly increases the concentration of responses on a small set of dominant regions, notably the United States and Japan. This effect was only marginally mitigated by additional instruction alignment, indicating that the process of defining and curating "correct" responses may inject cultural bias into the model's criteria. The findings have significant implications for the use of instruction-tuned models in global applications, where maintaining diverse cultural viewpoints could be critical.